Skip to content

Latest commit

 

History

History
32 lines (32 loc) · 935 Bytes

README.md

File metadata and controls

32 lines (32 loc) · 935 Bytes

1. Install the boto package for python

This will allow you to programatically interface with your AWS account

$ sudo pip install boto

Create a .boto file in your home directory

$ touch ~/.boto

Insert the following into .boto with your AWS credentials. These credentials will be used to access you AWS cluster as well as AWS's S3 storage.

[Credentials]
aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXX+XXXX

2. Clone repository

  • Place in home folder
  • Move into the HadoopInstallation folder
$ git clone https://github.com/aouyang1/HadoopInstallation.git
$ cd HadoopInstallation

Hadoop Installation

$ ./install_hadoop.sh ~/.ssh/<personal.pem> <region> <cluster-name>

You will need to SSH into the Namenode and start Hadoop after installation

$ $HADOOP_HOME/sbin/start-dfs.sh
$ $HADOOP_HOME/sbin/start-yarn.sh
$ $HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver