Maiar's Vault

This is a great tip and productivity enhancer.  You can now do native copy and paste with Tmux and OSX starting with Tmux 1.8 The guys over at thoughbot put this together and it's a good tip. You'll use the new copy-pipe  command to setup defaults in your tmux config. See the details in their post. http://robots.thoughtbot.com/tmux-copy-paste-on-os-x-a-better-future

Setting up a cluster can be time consuming especially with all the services: (Hbase, Zookeeper, Yarn, Hive, Oozie, Ambari, HCat, HDFS, WebHCat) Use this script and set it up with a single command. https://github.com/DemandCube/hadoop-single-node-cluster

Setup a hadoop 2 cluster with a single command: curl -sSL https://raw.githubusercontent.com/DemandCube/hadoop-single-node-cluster/master/INSTALL-HADOOP | bash -s -- -r For development purposes we wanted a easy way to setup an environment that's been tested everywhere we work: Vagrant AWS DigitalOcean This makes things much easier: Feel free to help add compatibility to other linux distros https://github.com/DemandCube/hadoop-single-node-cluster

Good read on HDFS small file compaction: With that decided, we then looked for options to aggregate and compact small files on Hadoop, identifying three possible solutions: filecrush  - a highly configurable tool by  Edward Capriolo  to “crush” small files on HDFS. It supports a rich set of configuration arguments and is available as a jarfile ( download it here ) ready to run on your cluster. It’s a sophisticated tool - for example, by default it won’t bother crushing a file which is within 75% of the HDFS block size already. Unfortunately, it does not work yet with Amazon’s s3:// paths, only hdfs:// paths - and our  pull request  to add this functionality is incomplete Consolidator  - a Hadoop file consolidation tool from the  dfs-datastores  library, written by  Nathan Marz . There is scant documentation for this - we could only find one paragraph,  in this email thread . It has fewer capabilities than filecrush, and could do w...

Download java jdk 7 with one line: wget --quiet --no-cookies -O /vagrant/jdk-7u45-linux-x64.rpm --header 'Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com;' http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.rpm --no-check-certificate creates=jdk-7u45-linux-x64.rpm

If your looking to see what address Kafka is bound to try using netstat. $netstat -tulpn

Files in a HDFS file system can be configured to save files to a trash dir.  If your just using it as a file store you need to manually clean up all the files.  You can do this with the following HDFS command: hadoop fs -Dfs.defaultFS=s3://myS3bucket -Dfs.trash.interval=0 -expunge

In hadoop 1.X HADOOP_HOME=/usr/lib/hadoop   Basically you'll use $HADOOP_HOME/bin/hadoop for your commands In hadoop 2.X   Basically you'll use $HADOOP_HOME/bin/hadoop for your commands plus /usr/lib/hadoop-hdfs/bin/hdfs /usr/lib/hadoop-mapreduce/bin/mapred /usr/lib/hadoop-yarn/bin/yarn Typical commands you'll want to use are: hdfs dfs -ls hdfs balancer mapred job -list yarn jar   

############### # Install nginx ############### # add the repo to Nginx export NGINX_REPO_FILE=/etc/yum.repos.d/nginx.repo touch $NGINX_REPO_FILE chmod 644 $NGINX_REPO_FILE chown root:root $NGINX_REPO_FILE echo "[nginx]" > $NGINX_REPO_FILE echo "name=nginx repo" >> $NGINX_REPO_FILE echo 'baseurl=http://nginx.org/packages/centos/$releasever/$basearch/' >> $NGINX_REPO_FILE echo "gpgcheck=0" >> $NGINX_REPO_FILE echo "enabled=1" >> $NGINX_REPO_FILE # Verify it worked cat $NGINX_REPO_FILE yum repolist #install nginx yum -y install nginx.x86_64 # config: /etc/nginx/nginx.conf # config: /etc/sysconfig/nginx # pidfile: /var/run/nginx.pid # User configs # /etc/nginx/conf.d/*.conf; # Log location # /var/log/nginx/access.log

Like most of my posts this is short To use distcp s3://key:secret@bucket/ you must have it setup as a file system configured with the NameNode.  So you'll basically always use it like s3://bucket-name/ s3 implementation here save blocks of files on hadoop and scrambles the names.  It can't be used standalone. If you want to use s3 standalone use s3n. You can test with hadoop fs -ls s3://bucket-name/ if you can access it great, it works. s3n - which stands for the s3 native protocol has a 5 Gig file size limitation of amazon. That's the short of it. -Steve

Short answer is that in Hadoop2 but S3 and S3N setup both in: "core-site.xml" # To Setup S3 Block Filesystem   fs.default.name   s3://BUCKET   fs.s3.awsAccessKeyId   ID   fs.s3.awsSecretAccessKey     SECRET # To Setup S3N Native Filesystem   fs.default.name   s3n://BUCKET   fs.s3n.awsAccessKeyId   ID   fs.s3n.awsSecretAccessKey     SECRET

My new favorite IDE for python is now iPython, been doing more work with scientific computing and machine learning which has lead me to discover iPython.  What a pleasure it is to work with and it's being developed at Berkeley right next door. If you like better interactivity, documentation, autocomplete and stack traces, just use iPython. Check it out:  http://ipython.org/

In case you didn't know the default java class path includes.  If you place jars here they will be found by default /Library/Java/Extensions/

erl -env ERL_LIBS "/my/path/to/module" export ERL_LIBS=/my/path/to/module erl -pa /my/path/to/module erlc -o /dir/to/save/compiled/file mymodule.erl NOTE:Modules must reside in a file with the same name as the file. NOTE:the lib path will not load if it doesn't contain a ebin dir Example:   /Users/joe/mymodule/ebin $ export ERL_LIBS=/Users/joe/mymodule; erl You can test to see if the path loaded with >code:get_path().

Must be a very well kept secret "BASH HAS A VI MODE". set -o vi set -o emacs How I have never come across this before is beyond me http://www.catonmat.net/blog/bash-vi-editing-mode-cheat-sheet/ http://www.catonmat.net/download/bash-vi-editing-mode-cheat-sheet.pdf

smorin$ ./mongod --dbpath ../../mongodata/ --help Allowed options: General options:   -h [ --help ]          show this usage information   --version              show version information   -f [ --config ] arg    configuration file specifying additional options   -v [ --verbose ]       be more verbose (include multiple times for more                          verbosity e.g. -vvvvv)   --quiet                quieter output   --port arg             specify port number   --bind_ip arg          comma separated list of ip addresses to listen on -                          all local ips by default   --logpath arg        ...

smorin$ ./bin/mongo --help MongoDB shell version: 1.8.1 usage: ./bin/mongo [options] [db address] [file names (ending in .js)] db address can be:   foo                   foo database on local machine   192.169.0.5/foo       foo database on 192.168.0.5 machine   192.169.0.5:9999/foo  foo database on 192.168.0.5 machine on port 9999 options:   --shell               run the shell after executing files   --nodb                don't connect to mongod on startup - no 'db address'                         arg expected   --quiet               be less chatty   --port arg            port to connect to   --host arg            server to connect to   --...

There isn't official documentation for the command line arguments as of redis-cli.  If you type --help for the command line arguments you get the following smorin$ ./redis-2.2.4//src/redis-cli --help redis-cli 2.2.4 Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]   -h    Server hostname (default: 127.0.0.1)   -p        Server port (default: 6379)   -s      Server socket (overrides hostname and port)   -a    Password to use when connecting to the server   -r      Execute specified command N times   -n          Database number   -x               Read last argument from STDIN   -d   Multi-bulk delimiter in for raw formatting (default: \n)   --raw            Use raw formatting for replies (default when STDOUT is not a tty)   --help     ...

There is only one argument for redis-server.  It's the path to the redis.conf file.  There is a example of this file in the redis src base.  There isn't a official documentation page as of 6/2011 $redis-server /path/to/redis/config/redis.conf

$ sbcl --eval '(progn (print *posix-argv*)(quit))' two three