Home

虚拟机集群安装

安装virtualbox

https://www.virtualbox.org/wiki/Downloads

安装vagrant

https://www.vagrantup.com/downloads.html

使用vagrant下载镜像

windows下进入cmd

mac下进入terminal

进入工作目录

cd work

vagrant box add centos6 https://github.com/CommanderK5/packer-centos-template/releases/download/0.6.7/vagrant-centos-6.7.box

vagrant init centos6

使用定制的Vagrantfile覆盖work下的文件

# -*- mode: ruby -*-

# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.define "hdp1" do |hdp1|
    hdp1.vm.box = "centos6"
    hdp1.vm.hostname = "hdp1"
    hdp1.vm.network "private_network", ip: "192.168.2.1"
    hdp1.vm.network :forwarded_port, host: 7180, guest: 7180
    config.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus = "1"
    end
  end

  config.vm.define "hdp2" do |hdp2|
    hdp2.vm.box = "centos6"
    hdp2.vm.hostname = "hdp2"
    hdp2.vm.network "private_network", ip: "192.168.2.2"
    config.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus = "1"
    end
  end  

  config.vm.define "hdp3" do |hdp3|
    hdp3.vm.box = "centos6"
    hdp3.vm.hostname = "hdp3"
    hdp3.vm.network "private_network", ip: "192.168.2.3"
    config.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus = "1"
    end
  end

  config.vm.define "hdp4" do |hdp4|
    hdp4.vm.box = "centos6"
    hdp4.vm.hostname = "hdp4"
    hdp4.vm.network "private_network", ip: "192.168.2.4"
    config.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus = "1"
    end
  end

end

启动vagrant

vagrant up

停止vagrant

vagrant halt

销毁vagrant

vagrant destroy

操作系统基本设置

mac: vagrant ssh hdp1

windows: 使用teraterm或其他的终端工具ssh hdp1(192.168.2.1):2222

关闭iptables

service iptables stop
chkconfig iptables off

关闭selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

字符编码统一

echo 'LANG="en_US.UTF-8"' > /etc/sysconfig/i18n 
echo 'SYSFONT="latarcyrheb-sun16"' >> /etc/sysconfig/i18n

同步时区时间

yum -y install rsync
rsync -a /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
yum install ntp -y
ntpdate 0.centos.pool.ntp.org
service ntpd start

添加hosts文件

/etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
192.168.2.1 hdp1
192.168.2.2 hdp2
192.168.2.3 hdp3
192.168.2.4 hdp4

免密码登录

把hdp1作为clouderamanager
在hdp1执行
ssh-keygen -t rsa(一直回车)
把/root/.ssh/id_rsa.pub复制到hdp1,hdp2,hdp3,hdp4并且改名为authorized_keys，修改权限为600
scp /root/.ssh/id_rsa.pub hdp1:/root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
scp /root/.ssh/id_rsa.pub hdp2:/root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
scp /root/.ssh/id_rsa.pub hdp3:/root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
scp /root/.ssh/id_rsa.pub hdp4:/root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

无效化hugepage

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled 
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

修改swappiness

echo "vm.swappiness = 0" >> /etc/sysctl.conf
sysctl -p

安装集群管理工具ansible

yum install epel-release -y
yum install ansible -y

更新yum后重启动

yum update -y
shutdown -r now

安装ClouderaManager

$ wget https://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

$ chmod u+x cloudera-manager-installer.bin

$ ./cloudera-manager-installer.bin

安装结束后打开
http://192.168.2.1:7180(hdp1的ip)
接下去按照向导安装hadoop

详细参考
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_a.html#cmig_topic_6_5

/etc/sysctl.d]# vi 80-hadoop.conf

Hadoop OS post instalation Kernel Tunning

vm.swappiness=0
net.ipv4.tcp_retries2=2
vm.overcommit_memory=1
net.core.somaxconn=4096
kernel.shmall=4294967296
kernel.shmmax=68719476736
kernel.sem="500 512000 64 2048"
kernel.panic=1
kernel.panic_on_oops=1
kernel.softlockup_panic=1
kernel.hung_task_panic=1
net.core.netdev_max_backlog=4096
net.core.rmem_max=8388608
net.core.wmem_max=8388608
net.ipv4.tcp_rmem="4096 87380 8388608"
net.ipv4.tcp_wmem="4096 87380 8388608"
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1