Ganglia Setup On Mac OSX

The platform is OS X Mountain Lion.

A couple of preparations:

Ganglia has so many components, configurations that makes setting it up on Mac OSX quite a task.
Using package tool would be a better choice than tar ball.

1. I upgraded XCode 4.5, since brew complained on my previous XCode 4.1. And make sure "Command Line Tool" is being installed from XCode.

2. Brew is good, and I actually blow away Mac Port for it. (They are conflicting each other... bumm).
Uninstall Mac Port.

sudo port -fp uninstall installed

sudo rm -rf \
    /opt/local \
    /Applications/DarwinPorts \
    /Applications/MacPorts \
    /Library/LaunchDaemons/org.macports.* \
    /Library/Receipts/DarwinPorts*.pkg \
    /Library/Receipts/MacPorts*.pkg \
    /Library/StartupItems/DarwinPortsStartup \
    /Library/Tcl/darwinports1.0 \
    /Library/Tcl/macports1.0 \
    ~/.macports

3. Install brew by running a single command:

ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"

4. Then install and config ganglia :

brew install ganglia

5. Generate config file :

gmond --default_config > /usr/local/etc/gmond.conf

6. Enable Apache and PHP. Then install additional php package for ganglia by running:

curl -s http://php-osx.liip.ch/install.sh | bash -

7. Download gnaglia-web tar ball, and install it.

8. Suppose you put ganglia-web at ~/Sites/ganglia, then there at that directory is a file "conf_default.php"

spot "/usr/bin/rrdtool" in that file and replace with "/usr/local/bin/rrdtool" or other path that rrdtool is installed. Then need to run "make install" again.

9. Start gmond and gmetad.

sudo gmetad --debug=1
sudo gmond --debug=3

10. Open ganglia web page:

localhost/~{username}/ganglia

Apache/PHP

Make localhost/~gfan/ accessible:

USER_DIR=$(basename $(echo ~))
sudo bash -c "cat > /etc/apache2/users/$USER_DIR.conf" <<TEXT
<Directory "/Users/$USER_DIR/Sites">
     Options Indexes MultiViews
     AllowOverride None
     Order allow,deny
     Allow from all
</Directory>
TEXT

Hive/Yarn Debug

localhost:8088   --check mapreduce progress.  --go into single MapReduce task to check syslog, which is a convenient way to check task log.

To use HBaseStorageHandler from Hive, got to have these dependencies in $HADOOP_HOME/shared/hadoop/mapreduce/

zookeeper-3.4.3-cdh4.0.0.jar
guava-11.0.2.jar
hive-common-0.8.1-cdh4.0.0.jar

hive-contrib-0.8.1-cdh4.0.0.jar
hive-exec-0.8.1-cdh4.0.0.jar

datanucleus-connectionpool-2.0.3.jar datanucleus-enhancer-2.0.3.jar
datanucleus-core-2.0.3.jar datanucleus-rdbms-2.0.3.jar

Hive Quick Note

1. open info log level:
bin/hive -hiveconf hive.root.logger=INFO,console

2. HIVE weird query result:
1> Where '='   comparing two things with different types. Would return result not error 

3. hive CLI 
echo "use bi_thunder; select * from tmp_kfb_task1_4;" | ./hv > ~/gfan/111

4. show create table table_name;will show the script that creates the table;

Manipulate Repository via SVN+SSH

Example to manipulate a directory in repo on server:
svn ls/co/copy/delete svn+ssh://gfan@www.serverip.com/path/to/directory -m "remove directory"