第1章Hadoop集群搭建 Ver1.5-20221122

Imagemap

hide

第1章Hadoop集群搭建
Ver1.5-20221122

hide

VMware上安装Hadoop3.x

hide

安装及配置虚拟机

hide

wmware安装与Linux安装

leaf

安装WMWare Workstation10，注册

hide

配置wmware的虚拟网络

leaf

编辑->虚拟网络编辑器

hide

添加网络->选择Nat模式->NAT设置

leaf

网关:192.168.137.2

leaf

选中:将主机虚拟适配器连接到此网络
选中:使用本地DHCP服务将IP地址分配给虚拟机

leaf

子网IP:192.168.137.0,子网掩码:255.255.255.0

hide

在WMWare上创建虚拟机

hide

创建master主机

leaf

选择默认的:典型

leaf

下一步，选择安装程序光盘映像文件->浏览->前面下载的文件:
CentOS-7-x86_64-Minimal-2009.iso

leaf

下一步，选中Linux操作系统，CentOS64位

leaf

下一步，设置虚拟机名称:CentOS_Hadoop3.x_master,选择位置

leaf

下一步，最大磁盘48G，将虚拟磁盘存储为单个文件

leaf

完成设置，开始安装

hide

远程连接虚拟机

hide

leaf

crt工具安装

leaf

crt工具的配置与连接虚拟机

leaf

以下linux操作如无说明均在crt里操作

hide

hide

修改显示主机名

leaf

hostnamectl set-hostname master

hide

修改网络主机名

hide

vi /etc/sysconfig/network

leaf

HOSTNAME=master

leaf

systemctl restart network

hide

leaf

说明:替换下面这些IP和名字后在命令行执行这些echo命令

leaf

echo '' > /etc/hosts
echo '192.168.137.100 master master.centos.com' >> /etc/hosts
echo '192.168.137.101 slave1 slave1.centos.com' >> /etc/hosts
echo '192.168.137.102 slave2 slave2.centos.com' >> /etc/hosts
echo '192.168.137.103 slave3 slave3.centos.com' >> /etc/hosts

hide

leaf

yum -y install ntp

hide

虚拟机安装Java

hide

leaf

yum -y install wget

leaf

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

leaf

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

hide

hide

rpm -qa|grep jdk
rpm -e --nodeps 刚查出来的jdk

leaf

[root@c23 ~]# rpm -qa|grep jdk
copy-jdk-configs-3.3-10.el7_5.noarch
java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64
java-1.8.0-openjdk-headless-1.8.0.342.b07-1.el7_9.x86_64
java-1.8.0-openjdk-devel-1.8.0.342.b07-1.el7_9.x86_64
[root@c23 ~]# rpm -e --nodeps java-1.8.0-openjdk-devel-1.8.0.342.b07-1.el7_9.x86_64
rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.342.b07-1.el7_9.x86_64
rpm -e --nodeps java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64
rpm -e --nodeps copy-jdk-configs-3.3-10.el7_5.noarch[root@c23 ~]# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.342.b07-1.el7_9.x86_64

[root@c23 ~]# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.342.b07-1.el7_9.x86_64
[root@c23 ~]# rpm -e --nodeps copy-jdk-configs-3.3-10.el7_5.noarch
[root@c23 ~]# rpm -qa|grep jdk
[root@c23 ~]#

leaf

运行结果如图

User Link

hide

leaf

sudo yum install -y java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64 --skip-broken

hide

hide

leaf

预期结果:
[root@c31 yum.repos.d]# java -version
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (build 25.322-b06, mixed mode)
[root@c31 yum.repos.d]#

hide

hide

leaf

预期结果最后一行可能包含如下路径:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre

hide

leaf

sudo yum install -y java-1.8.0-openjdk-devel-1.8.0.322.b06-1.el7_9.x86_64

hide

虚拟机安装Hadoop集群

hide

配置主机master

hide

下载安装hadoop3.3.1

leaf

mkdir /root/hadoop
cd /root/hadoop
wget https://mirrors.aliyun.com/apache/hadoop/core/hadoop-3.3.1/hadoop-3.3.1.tar.gz
tar -zxf hadoop-3.3.1.tar.gz -C /usr/local

hide

配置主机hadoop3.3.1

hide

在/usr/local/hadoop-3.3.1/etc/hadoop目录下修改配置

leaf

cd /usr/local/hadoop-3.3.1/etc/hadoop

hide

vi ./core-site.xml

leaf

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9864</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-3.3.1/tmp</value>
</property>
<property>
   <name>hadoop.native.lib</name>
   <value>false</value>
   <description>Should native hadoop libraries, if present, be used.
   </description>
</property>
</configuration>

hide

vi ./hadoop-env.sh

leaf

说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考java -verbose命令

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre

leaf

export HADOOP_HOME=/usr/local/hadoop-3.3.1

hide

vi ./hdfs-site.xml

leaf

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9868</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>

hide

vi ./mapred-site.xml

leaf

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>

hide

vi ./yarn-site.xml

leaf

<configuration>


<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
</configuration>

hide

vi ./yarn-env.sh

leaf

说明:在文件上方空白行处添加如下：

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre

hide

leaf

说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如下：

leaf

slave1
slave2
slave3

hide

vi ./hadoop-env.sh

leaf

说明:在刚才添加的export行后添加如下5行内容:

leaf

HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
HADOOP_SHELL_EXECNAME=root

hide

修改linux系统配置文件profile并使生效

hide

vi /etc/profile

leaf

说明:在文件最下方export行后添加如下：

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
export HADOOP_HOME=/usr/local/hadoop-3.3.1
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin

leaf

source /etc/profile

hide

验证hadoop相关配置是否生效

hide

leaf

说明：如果显示如下内容说明hadoop配置生效
[root@c31 ~]# hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /usr/local/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar

hide

克隆从机slave

hide

从master克隆出slave

hide

主机master关机

leaf

leaf

wmware左侧树图->右击主机master->管理->克隆->虚拟机当前状态->创建完整克隆,选择存放目录

hide

leaf

JPS无法查看已经启动的Java进程信息

hide

Failed to replace a bad datanode

hide

leaf

java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try.

hide

leaf

因为，我的集群共有3个DataNode节点，而我设置的默认副本数是3个。在执行写入到HDFS的操作时，当我的一台Datanode写入失败时，它要保持副本数为3,它就会去寻找一个可用的DataNode节点来写入，可是我的流水线上就只有3 台DataNode节点，所以导致会报错Failed to replace a bad datanode

hide

查看自己已经存在的副本数

leaf

Hadoop配置文件中的hdfs-site.xml

leaf

<property>
<name>dfs.replication</name>
<value>3</value>
</property>

hide

leaf

在hdfs-site.xml 文件中添加下面几行代码

leaf

<property>
<name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
<value>NEVER</value>
</property>

leaf

User Link

hide

hide

Byte、KB、MB、GB、
TB、PB、EB、ZB、
YB、NB、DB、CB、
XB

leaf

1B (Byte字节)；
1KB（Kilobyte） = 2^10 B = 1024 B；
1MB（Megabyte） = 2^10 KB = 1024 KB = 2^20 B；
1GB（Gigabyte） = 2^10 MB = 1024 MB = 2^30 B；
1TB（Terabyte） = 2^10 GB = 1024 GB = 2^40 B；
1PB（Petabyte） = 2^10 TB = 1024 TB = 2^50 B；
1EB（Exabyte） = 2^10 PB = 1024 PB = 2^60 B；
1ZB（Zettabyte） = 2^10 EB = 1024 EB = 2^70 B；
1YB（YottaByte） = 2^10 ZB = 1024 ZB = 2^80 B；
1BB（Brontobyte） = 2^10 YB = 1024 YB = 2^90 B；
1NB（NonaByte） = 2^10 BB = 1024 BB = 2^100 B；
1DB（DoggaByte） = 2^10 NB = 1024 NB = 2^110 B；
1CB (Corydonbyte) = 2^10 DB = 1024 DB = 2^120 B；
1XB (Xerobyte) = 2^10 CB = 1024 CB = 2^130 B；

hide

退出安全模式

hide

查看hdfs safe mode

leaf

hadoop dfsadmin -safemode get

hide

退出hdfs safe mode

leaf

hadoop dfsadmin -safemode leave

hide

降低hdfs上权限

hide

给根目录加权限

leaf

hdfs dfs -chmod -R 777 /

hide

HDFS重新格式化

leaf

1,删除所有从机的/data/hadoop目录:
rm -rf /data/hadoop

leaf

2,删除hadoop目录下的/tmp目录
rm -rf /usr/local/hadoop-3.3.1/tmp/*

leaf

3,停止集群
stop-all.sh

leaf

4,格式化hdfs
hdfs namenode -format

leaf

3,启动集群
start-all.sh

hide

Liux时区设置

leaf

使用cp,替换当前时区为上海时区，如有提示，输入yes进行确认
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime