hadoop版本：3.1.2 系统类型：ubuntu-server-16.04 3台 jdk版本：1.8
查看或修改主机名 sudo vim /etc/hostname 修改网络映射 sudo vim /etc/hosts 192.168.1.121 hadoop-master 192.168.1.122 hadoop-slave0 192.168.1.123 hadoop-slave1 删除以下网络映射 127.0.0.1 hadoop-master
添加用户组和用户 sudo addgroup hadoop sudo adduser -ingroup hadoop hadoop
配置Hadoop用户为管理员权限 sudo vim /etc/sudoers hadoop ALL=(ALL:ALL) ALL 切换用户 su hadoop 设置ssh免密登陆（主从机都要设置） ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh localhost exit
复制主机pubKey至从机 ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-slave0 ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-slave1
master上测试sleve连接 ssh 192.168.1.122 ssh 192.168.1.123 slave上测试master连接 ssh 192.168.1.121
(1)安装jdk 解压安装文件并移动到安装目录 tar -zxvf jdk-8u151-linux-x64.tar.gz sudo mv jdk-8u151-linux-x64 /usr/local/jdk1.8 配置环境变量 sudo vim /etc/profile export JAVA_HOME=/usr/local/jdk1.8 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=.:$PATH:${JAVA_HOME}/bin 文件修改生效 source /etc/profile 测试安装 java -version
(2)安装zookeeper 解压安装文件并移动到要安装目录 tar -xzvf zookeeper-3.4.9.tar.gz mv zookeeper-3.4.9 /usr/local/zookeeper 配置环境变量 sudo vim /etc/profile export ZOOKEEPER_HOME=/usr/local/zookeeper export PATH=$ZOOKEEPER_HOME/bin:$PATH source /etc/profile 配置zookeeper cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/usr/local/zookeeper/data clientPort=2181 server.1=hadoop-master:2888:3888 server.2=hadoop-slave0:2888:3888 server.3=hadoop-slave1:2888:3888 每个节点/usr/local/zookeeper/data目录下写入对应ID echo "1" > myid echo "2" > myid echo "3" > myid 启动zookeeper zkServer.sh start 查看各个zk节点状态 zkServer.sh status zk客户端连接并验证 zkCli.sh -server hadoop-master:2181 ls / quit 说明：配置部分三台机器都需要，可用同步命令 scp /etc/profile hadoop-slave0:/etc scp -r /usr/local/zookeeper hadoop-slave0:/usr/local/zookeeper (3)安装hadoop 解压安装文件并移动到要安装目录 tar -zxvf hadoop-3.1.2.tar.gz sudo mv hadoop-3.1.2 /usr/local/hadoop/hadoop-3.1.2 给hadoop安装目录设置用户操作权限 sudo chown -R hadoop:hadoop /usr/local/hadoop
配置环境变量 sudo vim /etc/profile export HADOOP_HOME=/usr/local/hadoop/hadoop-3.1.2 export PATH=$HADOOP_HOME/bin:$PATH export PATH=$PATH:$HADOOP_HOME/sbin 配置文件生效 source /etc/profile
(1)配置hadoop-env.sh文件 sudo vim /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/hadoop-env.sh 设置JAVA_HOME export JAVA_HOME=/usr/local/jdk1.8 配置生效 source /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/hadoop-env.sh (2)配置core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop/data</value> </property> <!--zookeeper配置 --> <property> <name>ha.zookeeper.quorum</name> <value>hadoop-master:2181,hadoop-slave0:2181,hadoop-slave1:2181</value> </property> (3)配置hdfs-site.xml <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.permissions.enabled</name> <value>false</value> </property> <!--指定hdfs的nameservice为ns1，需要和core-site.xml中的保持一致 --> <property> <name>dfs.nameservices</name> <value>ns1</value> </property> <!-- hadoop-master下面有两个NameNode，分别是nn1，nn2 --> <property> <name>dfs.ha.namenodes.ns1</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn1</name> <value>hadoop-master:9000</value> </property> <!-- nn1的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn1</name> <value>hadoop-master:9870</value> </property> <!-- nn2的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn2</name> <value>hadoop-slave0:9000</value> </property> <!-- nn2的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn2</name> <value>hadoop-slave0:9870</value> </property> <!-- 指定NameNode的edits元数据在JournalNode上的存放位置，可以配置地址或管理脚本 --> <!-- <value>file:///usr/local/hadoop/data/ha/ha-name-dir-shared</value> --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop-master:8485;hadoop-slave0:8485;hadoop-slave1:8485/ns1</value> </property> <!-- 指定JournalNode在本地磁盘存放数据的位置 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/usr/local/hadoop/data/ha/journalnode</value> </property> <!-- 开启NameNode失败自动切换 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 配置失败自动切换实现方式 --> <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行--> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <!-- 使用sshfence隔离机制时需要ssh免登陆 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value> </property> <!-- 配置sshfence隔离机制超时时间 --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> (4)配置mapred-site.xml <!-- 指定mr框架为yarn方式 --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> (5)配置yarn-site.xml <!-- 开启RM高可用 --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- 指定RM的cluster id --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yrc</value> </property> <!-- 指定RM的名字 --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- 分别指定RM的地址 --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>hadoop-master</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>hadoop-slave0</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>hadoop-master:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>hadoop-slave0:8088</value> </property> <!-- 指定zk集群地址 --> <property> <name>yarn.resourcemanager.zk-address</name> <value>hadoop-master:2181,hadoop-slave0:2181,hadoop-slave1:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- 是否启用日志聚合功能，日志聚合开启后保存到HDFS上 --> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <!-- 聚合后的日志在HDFS上保存多长时间，单位为s --> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>86400</value> </property> <!-- 启用自动恢复 --> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <!-- 制定resourcemanager的状态信息存储在zookeeper集群上 --> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property> <!-- 是否将对容器实施虚拟内存限制 --> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>5</value> </property> (6)workers文件中设置集群从机 hadoop-slave0 hadoop-slave1 (7)同步配置文件到其它服务器 scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/core-site.xml hadoop@hadoop-slave0:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/core-site.xml scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/core-site.xml hadoop@hadoop-slave1:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/core-site.xml
scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/hdfs-site.xml hadoop@hadoop-slave0:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/hdfs-site.xml scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/hdfs-site.xml hadoop@hadoop-slave1:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/hdfs-site.xml
scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/mapred-site.xml hadoop@hadoop-slave0:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/mapred-site.xml scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/mapred-site.xml hadoop@hadoop-slave1:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/mapred-site.xml
scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/yarn-site.xml hadoop@hadoop-slave0:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/yarn-site.xml scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/yarn-site.xml hadoop@hadoop-slave1:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/yarn-site.xml scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/workers hadoop@hadoop-slave0:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/workers scp -r /usr/local/hadoop/hadoop-3.1.2/etc/hadoop/workers hadoop@hadoop-slave1:/usr/local/hadoop/hadoop-3.1.2/etc/hadoop/workers
(1)启动各个节点zookeeper zkServer.sh start 查看各个zk节点状态 zkServer.sh status (2)启动hadoop-master hadoop-slave0 hadoop-slave1 的journalnode hdfs --daemon start journalnode 或 hadoop-daemon.sh start journalnode (3)在hadoop-master或hadoop-slave0上启动namenode（这里选择在hadoop-master上启动） namenode格式化(只第一次启动时执行一次) hdfs namenode -format 启动当前namenode hadoop-daemon.sh start namenode hdfs --daemon start namenode 同步namenode数据(hadoop-slave0上操作) hdfs namenode -bootstrapStandby
验证同步 cat /usr/local/hadoop/data/dfs/name/current/VERSION
(4)zk格式化（hadoop-master执行，第一次启动执行一次） hdfs zkfc -formatZK zk客户端连接并验证 zkCli.sh -server hadoop-master:2181 ls / ls /hadoop-ha quit
(5)启动集群 start-dfs.sh 启动master的datanode（master节点datanode未启动，可能workers中要增加主节点） hdfs --daemon start datanode start-yarn.sh jps查看各个节点启动情况
(6)验证 获取namenode节点状态 hdfs haadmin -getServiceState nn1 hdfs haadmin -getServiceState nn2
如果两台机器都为standby，则强制切换namenode节点nn1为active hdfs haadmin -transitionToActive --forcemanual nn1
(7)web页面访问 http://hadoop-master:9870/ http://hadoop-slave0:9870/
启动namenode hdfs --daemon start namenode hdfs --daemon start secondarynamenode datanode节点宕机后重启 hdfs --daemon start datanode yarn --daemon start nodemanager yarn --daemon start resourcemanager
开启防火墙 sudo ufw enable
关闭防火墙 sudo ufw disable
查看防火墙状态 sudo ufw status
检查端口监听情况 netstat -an | grep 9000
