hadoop集群搭建前的准备(一定要读):/14048416/2341450
hadoop分布式集群搭建:
1. 集群规划:
2.具体步骤:
(1)上传安装包,并解压到相关目录:
[hadoop@hadoop01 home]$ put c:/hadoop-2.6.5-centos-6.7.tar.gz
[hadoop@hadoop01 home]$tar -zxvf hadoop-2.6.5-centos-6.7.tar.gz -C /home/hadoop/apps
(2)修改配置文件:
[hadoop@hadoop01 home]$ cd /home/hadoop/apps/hadoop-2.6.5/hadoop/etc
这里需要修改6个配置文件:
hadoop-env.sh:
加入:export JAVA_HOME=/usr/java/jdk1.8.0_73
core-site.xml:
<property><name>fs.defaultFS</name><value>hdfs://hadoop01:9000</value></property><property><name>hadoop.tmp.dir</name><value>/home/hadoop/hadoopdata</value></property>
hdfs-site.xml:
<property><name>dfs.namenode.name.dir</name><value>/home/hadoop/hadoopdata/name</value><description>为了保证元数据的安全一般配置多个不同目录</description></property><property><name>dfs.datanode.data.dir</name><value>/home/hadoop/hadoopdata/data</value><description>datanode 的数据存储目录</description></property><property><name>dfs.replication</name><value>2</value><description>HDFS 的数据块的副本存储个数</description></property><property><name>dfs.secondary.http.address</name><value>hadoop02:50090</value><description>secondarynamenode 运行节点的信息,和 namenode 不同节点</description></property>
mapred-site.xml:
<property><name>mapreduce.framework.name</name><value>yarn</value></property>
yarn-site.xml:
<property><name>yarn.resourcemanager.hostname</name><value>hadoop03</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>YARN 集群为 MapReduce 程序提供的 shuffle 服务</description></property>
slaves:
hadoop01hadoop02hadoop03
(3)分发安装包到各个节点,Hadoop 集群的每个节点都需要安装 Hadoop 安装包:
[hadoop@hadoop01 etc]$scp -r hadoop-2.6.5 hadoop02:$PWD
[hadoop@hadoop01 etc]$scp -r hadoop-2.6.5 hadoop03:$PWD
(4)配置hadoop的环境变量
[hadoop@hadoop01 etc]$sudo vim /etc/profile:
加入:
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.5/
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[hadoop@hadoop01 etc]$source /etc/profile
(5)在 HDFS 主节点上执行命令进行初始化 namenode
[hadoop@hadoop01 etc]$hadoop namenode -format
(6)在 HDFS 上的主节点启动 HDFS
[hadoop@hadoop01 etc]$start-dfs.sh
(7)在 HDFS 上的主节点启动 YARN
[hadoop@hadoop01 etc]$sbin/start-yarn.sh
补充:
hdfs的web界面是: http://hadoop01:50070
yarn的web界面是: http://hadoop03:8088
查看集群的状态:hdfs dfsadmin -report
1、启动 namenode 或者 datenode
sbin/hadoop-daemon.sh start datanode
sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start secondarynamenode
2、启动 yarn nodemanager
sbin/yarn-daemon.sh start nodemanager
sbin/yarn-daemon.sh start resourcemanager