hadoop2.6.5集群搭建

1、准备三台centos7机器,这里我使用的vmware创建虚拟机,并在/etc/hosts配置host映射

192.168.8.144 master
192.168.8.145 slave1
192.168.8.146 slave2

2、配置ssh免密码登录
master执行

ssh_keygen -t dsa
cat /root/.ssh/id_dsa.pub >> /root/.ssh/authorized_keys
scp /root/.ssh/authorized_keys slave1:/root/.ssh
scp /root/.ssh/authorized_keys slave2:/root/.ssh

ssh slave1
ssh slave2

3、下载官网的jdk1.8、scala2.12.10、hadoop2.6.5、hbase1.2.2、hive1.2.2安装包并上传到centos7的master

scp -r jdk1.8 root@master：/Java
scp -r scala2.12.10 root@master：/Java
scp -r hadoop2.6.5 root@master：/Java
scp -r hbase1.2.2 root@master：/Java

4、master\slave1\slave2分别安装jdk1.8
安装vim命令

yum -y install vim*
vim /etc/profile

添加

export JAVA_HOME=/Java/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

执行

source /etc/profile

激活配置

5、关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

6、修改hadoop/etc/hadoop/hadoop-env.sh和yarn-env.sh,设置为实际${JAVA_HOME}安装路径

export JAVA_HOME=/Java/jdk.18

7、修改core-site.xml
首先创建如下目录

mkdir -p /hadoop/hdfs/tmp

添加

<configuration>
	<property>
			<name>fs.defaultFS</name>
			<value>hdfs://master:9000</value>
	</property>
	<property>
			<name>hadoop.tmp.dir</name>
			<value>/hadoop/hdfs/tmp</value>
	</property>
</configuration>

8、修改hdfs-site.xml

<configuration>
	<property>
			<name>dfs.replication</name>
			<value>2</value>
	</property>
</configuration>

9、修改mapred-site.xml

复制

cp mapred-site.xml.template mapred-site.xml

<configuration>
	<property>
			<name>mapreduce.framework.name</name>
			<value>yarn</value>
	</property>
</configuration>

10、修改yarn-site.xml

<configuration>
	<property>
			<name>yarn.nodemanager.aux-services</name>
			<value>mapreduce_shuffle</value>
	</property>
	<property>
			<name>yarn.resourcemanager.address</name>
			<value>master:8040</value>
	</property>
	<property>
			<name>yarn.resourcemanager.scheduler.address</name>
			<value>master:8030</value>
	</property>
	<property>
			<name>yarn.resourcemanager.resource.tracker.address</name>
			<value>master:8025</value>
	</property>
	<property>
			<name>yarn.resourcemanager.admin.address</name>
			<value>8141</value>
	</property>
	<property>
			<name>yarn.resourcemanager.webapp.address</name>
			<value>master:8088</value>
	</property>
</configuration>

11、创建文件slaves

touch /hadoop/hadoop2.6.5/etc/hadoop/slaves

添加

slave1
slave2

12、将修改后的hadoop安装包同步给slave1、slave2

scp -r hadoop-2.6.5 slave1:/hadoop
scp -r hadoop-2.6.5 slave2:/hadoop

13、修改所有节点的环境变量

vim /etc/profile

添加

export HADOOP_HOME=/hadoop/hadoop-2.6.5
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

14、在master节点格式化NameNode

hdfs namenode -format

15、master节点启动

start-all.sh

16、测试hadoop集群是否搭建成功
访问

master:50070
master:8088

查看是否正常

转载：https://blog.csdn.net/weixin_44099288/article/details/102071927