Hive/Yarn Debug

localhost:8088   --check mapreduce progress.  --go into single MapReduce task to check syslog, which is a convenient way to check task log.

To use HBaseStorageHandler from Hive, got to have these dependencies in $HADOOP_HOME/shared/hadoop/mapreduce/

zookeeper-3.4.3-cdh4.0.0.jar
guava-11.0.2.jar
hive-common-0.8.1-cdh4.0.0.jar

hive-contrib-0.8.1-cdh4.0.0.jar
hive-exec-0.8.1-cdh4.0.0.jar

datanucleus-connectionpool-2.0.3.jar datanucleus-enhancer-2.0.3.jar
datanucleus-core-2.0.3.jar datanucleus-rdbms-2.0.3.jar

Hive Quick Note

1. open info log level:
bin/hive -hiveconf hive.root.logger=INFO,console

2. HIVE weird query result:
1> Where '='   comparing two things with different types. Would return result not error 

3. hive CLI 
echo "use bi_thunder; select * from tmp_kfb_task1_4;" | ./hv > ~/gfan/111

4. show create table table_name;will show the script that creates the table;

Manipulate Repository via SVN+SSH

Example to manipulate a directory in repo on server:
svn ls/co/copy/delete svn+ssh://gfan@www.serverip.com/path/to/directory -m "remove directory"

Use VIP to workaround Hadoop/HBase localhost

In some cases, we want to use a dedicated IP address for pseudo-cluster instead of "localhost", for example, when your show-off machine changes IP address timely.

I'm not sure this would help because I didn't test by changing ip. Please give any comment you have.

I created a vip at a random address 192.168.1.130:

sudo ifconfig en1 inet 192.168.1.130 netmask 255.255.255.255 alias

I substitue localhost for all config files under my hadoop folder: 

find . -name *.xml -exec sed -i "" 's/localhost/192.168.1.130/g' {} \;

And I just started hbase/dfs and inserted some data,  and they work fine using that vip. 

And there created 

/hbase/.logs/192.168.1.130,60020,1346102408672 

using the vip.

To bring the interface down:

sudo ifconfig en1 inet down

Notice that: Max OSX sed has slightly different syntax than Linux, the empty double quote "" means replacing original file with output file.

Hadoop Cluster Setup, SSH Key Authentication

First from your “master” node check that you can ssh to the localhost without a passphrase:

$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t dsa -P “” -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

On your master node try to ssh again (as the hadoop user) to your localhost and if you are still getting a password prompt then.

$ chmod go-w $HOME $HOME/.ssh
$ chmod 600 $HOME/.ssh/authorized_keys
$ chown `whoami` $HOME/.ssh/authorized_keys

Now you need to copy (however you want to-do this please go ahead) your public key to all of your “slave” machine (don’t forget your secondary name node).  It is possible (depending on if these are new machines) that the slave’s hadoop user does not have a .ssh directory and if not you should create it ($ mkdir ~/.ssh)

$ scp ~/.ssh/id_dsa.pub slave1:~/.ssh/master.pub

Now login (as the hadoop user) to your slave machine.  While on your slave machine add your master machine’s hadoop user’s public key to the slave machine’s hadoop authorized key store.

$ cat ~/.ssh/master.pub >> ~/.ssh/authorized_keys

Now, from the master node try to ssh to slave.

$ssh slave1

If you are still prompted for a password (which is most likely) then it is very often just a simple permission issue.  Go back to your slave node again and as the hadoop user run this

$ chmod go-w $HOME $HOME/.ssh
$ chmod 600 $HOME/.ssh/authorized_keys
$ chown `whoami` $HOME/.ssh/authorized_keys

Try again from your master node.

$ssh slave1

And you should be good to go. Repeat for all Hadoop Cluster Nodes.

If run on a localhost which restarts frequently, change "hadoop.tmp.dir" in core-site.xml

<property>

    <name>hadoop.tmp.dir</name>

    <value>/var/hadoop</value>

    <final>true</final>

  </property>

From http://allthingshadoop.com/2010/04/20/hadoop-cluster-setup-ssh-key-authentication/

MRUNIT

1. Let CDH4 work with MRUNIT

<dependency>

<groupId>org.apache.mrunit</groupId>

<artifactId>mrunit</artifactId>

<version>0.9.0-incubating</version>

<classifier>hadoop2</classifier>

</dependency>

reference: https://blogs.apache.org/mrunit/entry/apache_mrunit_0_9_0#comment-1337534033000.

Quick Note

1.   p7zip is the Unix port for 7-zip. So use "sudo port install p7zip" to install