ElasticSearch

1. ElasticSearch transfers data scoll size setup. Will transfer up to 100 documents starting from the first one.

_search?from=0&size=100

Bash Looping

#times of try
times=36
#interval in seconds
interval=450

if [ $(date +%u) -eq 1 ]
then
 times=108
fi

echo "check if /data/hive/insights/brand/irm-free-insights-pipeline/${date}/_SUCCESS exits."
i=0
while [ $i -lt $times ]
do
s=$(ssh insights@jobs-aa-sched1 "(hadoop fs -ls /data/hive/insights/brand/irm-free-insights-pipeline/${date} | grep _SUCCESS | wc -l)")
if [ $s -eq 1 ]
then
  echo "perks pipeline success."
  exit 0
else
  echo "sleep for $interval seconds and check again, tried $i out of $times"
  sleep $interval
fi
i=$(expr $i + 1)
done

echo "insights pipeline failed."
exit 1

Hubot Lock Structure

1. Two kinds of locks :

"Uploading Lock"  single node in zookeeper

"Health"  with two children "targetingA" and "targetingB"

2. Whenever uploading (indexing) one cluster

We acquire "Uploading Lock" and set "Health/targetingA"  false. API cannot read from targetingA any more.

If you set "Health/targetingB" true (we call this "overwrite to targetingB"), that allows API read from targetingB

3. "Uploading Lock" make sure anytime, only one cluster is doing uploading.

"Health" lock mark the one API can use.

Hive Several Problem

1.
insert overwrite table foo
select a.*
from
(select c, d, e from too1
union all
select c, d, e from too2
) a


This wouldn't work if the column sequence in foo is other than c, d, e
Hive will map data to wrong column based on the sequence. (if only the column type match)

2.
select * form table where id not in ('a', 'b', 'c');

The records filtered out are not only id = 'a' 'b' 'c', also include null
if id is null. the record will also be filtered out.

3.
avoid group by too many columns, especially long length strings, slow down the speed and easy to get error when processing the row. Use group by id1, id2, id3 and use group_first(other column)

Backfill HBase Data

put 'perk_benchmark_complete', '51c0a5a5e4b04e95568bc531', 'c:q', '{"benchmark":{"score":{"other":25.73531446900622,"self":63.61400714763741},"contentCreated":{"other":39.478001720716996,"self":181.0},"contentCreatedPerUser":{"other":0.04595809280642258,"self":0.21071012805587894},"networkSize":{"other":175.5376223347399,"self":1898.0},"totalTrueReach":{"other":933052.3242595217,"self":7167783.878724266},"totalImpressions":{"other":1925504.5258158229,"self":1.0672166152029522E7},"trueImpressions":{"other":1925504.5258158229,"self":1.0672166152029522E7}},"brandComparison":{"size":{"peer":-1.0,"self":-1.0},"averageScore":{"peer":-1.0,"self":-1.0},"averageContent":{"peer":-1.0,"self":-1.0},"feedback":{"peer":-1.0,"self":-1.0}}}'


deleteall 'perk_benchmark_complete', '51c0a5a5e4b04e95568bc531'


scan 'perk_benchmark_complete', {STARTROW=>'51c0a5a5e4b04e95568bc531', LIMIT=>3}

Bloom Filter and Distributed Map

insert overwrite local directory 'delisted_user_${networkAbbr}' select ks_uid , delist_date from delist_user where dt=${dateString} ...

4:45 PM (select * from collect_moment_contrib_view_${networkAbbr} where ! bloom_contains(concat( cast(ks_uid as string), "_", content_id), distributed_bloom( 'dup_moment_bloom_${networkAbbr}')) and ! bloom_contains( cast(ks_uid as string), distributed_bloom( 'optout_bloom_${networkAbbr}')) and distributed_map( ks_uid, "delisted_user_${networkAbbr}" ) is null

4:46 PM insert overwrite local directory 'dup_moment_bloom_${networkAbbr}' select bloom( concat(cast(ks_uid as string), "_", content_id) ) from duplicate_moments where dt = ${dateString} and network_abbr = "${networkAbbr}" and label = "DUPLICATE" ; add file dup_moment_bloom_${networkAbbr};

Distcp between CDH3 and CDH4

Assume aa is CDH3 and dev is CDH4:

Run this on jobs-aa
hadoop distcp hftp://jobs-aa-hnn/root/my/dir hdfs://jobs-dev-hnn/root/my/dir

Reverse direction:


hadoop distcp hdfs://jobs-dev-hnn:50070/root/my/dir hdfs://jobs-aa-hnn/root/my/dir
(got to have this port number because CDH4 requires that)