Remove a region from HBase

Remove a region from HBase

Recently one of our hbase experienced outrage due to disk space full. Adding new nodes and balancing the cluster takes time.

After hdfs balancing with :

sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 65536000

sudo -u hdfs hdfs balancer -threshold 5

It gets back and functional. But when it runs MapReduce job from snapshot, it produces duplicate records in output. 

After digging into it a bit, I realized there are regions in HBase that are not online (shown in HBase UI), but since they are still residing in meta table and on the disk, the MapReduce job still running against them.

To remove those regions from HBase :

Find those regions from hbase:meta table

echo "scan 'hbase:meta'"| hbase shell &> dump.txt

And remove them:

deleteall 'hbase:meta', 'The name of the region'
Then run "hbase hbck PlatformData -repair" a couple times, until regions' data consistent.
Also find and remove the regions' file from hdfs :
sudo -u hdfs hadoop fs -rmr /hbase/data/default/user_profile/<regions hash>
Then the MapReduce job is good to go.

Use scutil to set host name

scutil --set HostName "localhost"

scutil --get HostName 

Note: make sure "localhost" is set in your /etc/hosts file

HBase Snapshot Restrictions

HBase Snapshot has two restrictions :

1. If in any case, regions got merged (usually manually) after the snapshot got taken, the snapshot would be invalid. Splitting regions are fine to snapshot.

2. If you restore data to create a table from snapshot, the replication on another cluster of that new table won't guarantee the data is integral.

Setup local dev env

apply the pitch for os x

uname -a

copy the ip and put in /etc/hosts as an alias of localhost

setup dns servers in system preference -> network -> advanced -> dns



Run Service in debug mode

gradle :apps:[Service_Name]:run --debug-jvm -Pdebug

Gradle Notes

1. To show cmd options of task "test"

 gradle help --task :test

Find class in jar

in lib do:

grep package/or/classname *

result :

abc.jar

look thru classes within it:

less abc.jar


shift+g : to to bottum
shift+n : go backward
space: go forward page by page

HBase - advanced configuration on column family level

HBase - advanced configuration on column family level

Block Size
HFile block size.Default is 64k,If you want to good sequential scan performance,it;s better to have larger  block size.
        Setting is during table creation
        hbase(main):002:0> create 'mytable',   {NAME => 'colfam1', BLOCKSIZE => '65536'}
        Or with code
        On HColumnDescriptor there is a method : setBlocksize(int)

Block Cache
        You can block cache for specific column family in order to improve caching for other column families for example.

hbase(main):002:0> create 'mytable',
{NAME => 'colfam1', BLOCKCACHE => 'false’}

Aggresive caching

       You can choose column families to be in highter priority for caching.

       hbase(main):002:0> create 'mytable',
       {NAME => 'colfam1', IN_MEMORY => 'true'}

Bloom filters

       You enable bloom filters on the column family, like this:
       hbase(main):007:0> create 'mytable',
       {NAME => 'colfam1', BLOOMFILTER => 'ROWCOL'}

       A row-level bloom filter is enabled with ROW, and a qualifier-level bloom filter is enabled with ROWCOL

 TTL
       By defining Time To Live on some column family will delete the data after given amount  of time
       Example:
      hbase(main):002:0> create 'mytable', {NAME => 'colfam1', TTL => '18000'}

     Data in colfam1 that is older than 5 hours is deleted during the next major compaction.

Compression

    Compression defenition impacts HFiles and their data. This can save disk I/O and instead pay for higher CPU utilization.
    hbase(main):002:0> create 'mytable',
    {NAME => 'colfam1', COMPRESSION => 'SNAPPY'}

Cell versioning

    By default 3 versions of values are saved. Can be changed

    hbase(main):002:0> create 'mytable', {NAME => 'colfam1', VERSIONS => 5,
    MIN_VERSIONS => '1'}

Ops

1. disk info : df
io info : iostat
usage info: free
cpu info: top / htop (is pretty cool)
web io: iftop

Redis Note

1) remove all keys with prefix
for key in `echo 'KEYS sectionId_*' | redis-cli | awk '{print $1}'`
 do echo DEL $key
done | redis-cli

Reset ttl
for key in `echo 'KEYS *' \| redis-cli \| awk '{print $1}'`
do
echo expire $key 86400
done | redis-cli

Edit Bash Prompt

Edit .bash_profile:

Between the quotation marks, you can add the following lines to customize your Terminal prompt:

• \d – Current date
• \t – Current time
• \h – Host name
• \# – Command number
• \u – User name
• \W – Current working directory (ie: Desktop/)
• \w – Current working directory, full path (ie: /Users/Admin/Desktop)

So, let’s say you want your Terminal prompt to display the User, followed by the hostname, followed by the directory, the .bashrc entry would be:

export PS1="\u@\h\w$ "

Generic HBase Export

code/analytics/etl/bing/bing-assembly/src/assembly/oozie/generic_hbase_export.xml

com.klout.bing.hbaseexport.HbaseTableExportMapper

Convert word doc to md file

textutil -convert html ~/Downloads/UniquereachImpressionBreakdown.docx  -stdout | pandoc -f html -t markdown -o output.md

Bash Format and Send Email

printTable.sh

#!/bin/bash
#
#   Script that output a hive table to a file

if [ $# -ne 3 ]
then
  echo "Script that output a hive table to a file."
  echo "Usage: $0 <db_name> <table_name> <output_path> "
  exit 1
fi

DB_NAME=$1
TABLE_NAME=$2
OUTPUT_PATH=$3

echo "use $1; select * from $TABLE_NAME;" | /home/insights/insights/hive/hv | tail -n+3 | head -n-1 > $OUTPUT_PATH


if [ "$email" != "email" ]
then
for line in $(echo $email | tr ',' '\n')
do

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no abc@host1 "(cd insights/bin; sh printTable.sh bi_insights tmp_kfb_user_meta ~/gfan/kfb_user_meta_$date.tsv && echo -e "brand_ks_uid\thandle\tbrand\temail\ttw_followers\tfp_likes\tnum_activities_90days\tcity\tstate\tcountry\tkdc_created_at" | cat - ~/gfan/kfb_user_meta_$date.tsv > ~/gfan/tmp && mv ~/gfan/tmp ~/gfan/kfb_user_meta_$date.tsv && zip ~/gfan/kfb_user_meta_$date.zip ~/gfan/kfb_user_meta_$date.tsv && echo 'Data Format: brand_ks_uid | handle | brand | email | tw_followers | fp_likes | num_activities_90days | city | state | country | kdc_created_at' | mutt -s 'KFB User Meta' $line -a ~/gfan/kfb_user_meta_$date.zip)"

done
fi

JUnit Test Note

Test on CL:

java -cp /Users/guangle/.m2/repository//junit/junit/4.8.1/junit-4.8.1.jar:target/thunder-libs-0.0.71.jar org.junit.runner.JUnitCore com.klout.thunder.common.UrlSummaryNormalizerTest

hbase note

1.  pre-split regions

rm /tmp/region.splits; for i in $(seq 1 1 99); do printf "%02d00\n" $i >> /tmp/region.splits ; done

create 'networkProfile3', {NAME => 'c', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAPPY', MIN_VERSIONS => '1', TTL => '7776000', BLOCKSIZE => '65536', IN_MEMORY => 'false ', BLOCKCACHE => 'true'}, {SPLITS_FILE => '/tmp/region.splits'}


2. flush sharded redis server

 for i in $(seq 1 7)
 do 
 host="flredis0"$i"-mini.private"  
 ssh $host '(echo "flushdb" | redis-cli)'

 done

Scala Notes

1. A function calling async function should return a future too.

def foo (b: B) : Future[A] {
    val c = aync {
       ...
    }

2. access a future:

 for {
      s <- c
   } yield {
      ...
   }
}

3. change a list of future to future of list
Future.sequence(x)

4. wrap anything with Future:
sync(None)

Test udf

 java -cp ./insights-etl-0.7.6.jar:~/insights/lib/*:/usr/lib/hive/lib/* com.klout.perk.GnipTupleUDTF

Hive Mapside Join Configuration

Disable mapside join

set hive.auto.convert.join=false;

Other configuration : go to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

search mapside join

Scala

Means case class returns not single types in different branches:

 Cannot prove that Product with Serializable <:< (T, U).

ElasticSearch

1. ElasticSearch transfers data scoll size setup. Will transfer up to 100 documents starting from the first one.

_search?from=0&size=100