Oozie job weird error : No input path Specified

Today I debugged with a couple guys on a strange oozie error. The MapRed job with error
"No input path Specified"
But we have the input dir set up in configuration and workflow.
It turned out to be that we missed two configurations for oozie job to tell oozie use the new api :

<property>
                    <name>mapred.mapper.new-api</name>
                    <value>true</value>
                </property>
                <property>
                    <name>mapred.reducer.new-api</name>
                    <value>true</value>
                </property>

Bash Quich Note

1. Each line in a file:

for line in $(cat 2); do echo $line; done;

2. Sed
prefer to use '|' as delimiter if possible:

sed 's|my/home/directory||g' < in > out

in place replacement :
sed -i 's|analytics/etl/maxwell/src/assembly/hive/maxwell/||g' in


3. Sed replace \n to ,\n

sed ':a;N;$!ba;s/\n/,\n/g'

4. sort by column

sort -t "," -k 2 -n input.csv
sorted by column 2

5. bash loop in number

for i in $(seq 0 855)
do
date=$(date --date "$i day ago" "+%Y%m%d")
echo "alter table oauth_user_services add if not exists partition (dt = '$date') location '$date';" >> partitions
done

HBase Lock and Override

HBase lock is like gate keeper.

Before bulkloading, set HBase lock first, then set HBase Override.

After bulkloading, release Override first and then unlock HBase.

Remove leading and tailing spaces of each line

cat input.txt | sed 's/^[ \t]*//;s/[ \t]*$//' > output.txt

SSH Key

ssh-keygen
With no passphrase
Keep the .ssh permission 700
Keep the .ssh/id_rsa permission 600

Hive table for lzo.deflated files

Should be able to say
stored by textfile, but cann't select * from where partition = ...

And when you select, must force hive to run a mapreduce job. For example select count(*)

Oozie Framework Improvement

• Use coordinator
• Hue
• upgrade to CHD4 (job name etc.)