Convert word doc to md file

textutil -convert html ~/Downloads/UniquereachImpressionBreakdown.docx  -stdout | pandoc -f html -t markdown -o output.md

Bash Format and Send Email

printTable.sh

#!/bin/bash
#
#   Script that output a hive table to a file

if [ $# -ne 3 ]
then
  echo "Script that output a hive table to a file."
  echo "Usage: $0 <db_name> <table_name> <output_path> "
  exit 1
fi

DB_NAME=$1
TABLE_NAME=$2
OUTPUT_PATH=$3

echo "use $1; select * from $TABLE_NAME;" | /home/insights/insights/hive/hv | tail -n+3 | head -n-1 > $OUTPUT_PATH


if [ "$email" != "email" ]
then
for line in $(echo $email | tr ',' '\n')
do

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no abc@host1 "(cd insights/bin; sh printTable.sh bi_insights tmp_kfb_user_meta ~/gfan/kfb_user_meta_$date.tsv && echo -e "brand_ks_uid\thandle\tbrand\temail\ttw_followers\tfp_likes\tnum_activities_90days\tcity\tstate\tcountry\tkdc_created_at" | cat - ~/gfan/kfb_user_meta_$date.tsv > ~/gfan/tmp && mv ~/gfan/tmp ~/gfan/kfb_user_meta_$date.tsv && zip ~/gfan/kfb_user_meta_$date.zip ~/gfan/kfb_user_meta_$date.tsv && echo 'Data Format: brand_ks_uid | handle | brand | email | tw_followers | fp_likes | num_activities_90days | city | state | country | kdc_created_at' | mutt -s 'KFB User Meta' $line -a ~/gfan/kfb_user_meta_$date.zip)"

done
fi

JUnit Test Note

Test on CL:

java -cp /Users/guangle/.m2/repository//junit/junit/4.8.1/junit-4.8.1.jar:target/thunder-libs-0.0.71.jar org.junit.runner.JUnitCore com.klout.thunder.common.UrlSummaryNormalizerTest

hbase note

1.  pre-split regions

rm /tmp/region.splits; for i in $(seq 1 1 99); do printf "%02d00\n" $i >> /tmp/region.splits ; done

create 'networkProfile3', {NAME => 'c', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAPPY', MIN_VERSIONS => '1', TTL => '7776000', BLOCKSIZE => '65536', IN_MEMORY => 'false ', BLOCKCACHE => 'true'}, {SPLITS_FILE => '/tmp/region.splits'}


2. flush sharded redis server

 for i in $(seq 1 7)
 do 
 host="flredis0"$i"-mini.private"  
 ssh $host '(echo "flushdb" | redis-cli)'

 done

Scala Notes

1. A function calling async function should return a future too.

def foo (b: B) : Future[A] {
    val c = aync {
       ...
    }

2. access a future:

 for {
      s <- c
   } yield {
      ...
   }
}

3. change a list of future to future of list
Future.sequence(x)

4. wrap anything with Future:
sync(None)

Test udf

 java -cp ./insights-etl-0.7.6.jar:~/insights/lib/*:/usr/lib/hive/lib/* com.klout.perk.GnipTupleUDTF

Hive Mapside Join Configuration

Disable mapside join

set hive.auto.convert.join=false;

Other configuration : go to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

search mapside join