第2章Hadoop集群搭建

Imagemap
第2章Hadoop集群搭建任务前言学习目标（1）掌握安装配置虚拟机。（2）掌握如何在本地和Linux虚拟机中安装JDK。（3）掌握搭建Hadoop完全分布式集群环境。任务背景集群环境分类单机版一台机器上运行Hadoop伪分布式一台单机上模拟组建的多节点集群完全分布式是在多台单机上组建的分布式集群任务指导http://bigdata.hddly.cn/b37066/B37066L02 ...学习视频在笔记本上安装hadoop3.x环境概述与环境准备http://i.hddly.cn/media/G5dykd0RWL.mp4安装及创建虚拟机http://i.hddly.cn/media/HUSTaLf3ok.mp4http://i.hddly.cn/media/IrlUMiUIuu.mp4配置虚拟机http://i.hddly.cn/media/MDQuzpBM4A.mp4http://i.hddly.cn/media/rUvi0fQnOj.mp4虚拟机安装Javahttp://i.hddly.cn/media/SiF5MekLLr.mp4虚拟机安装Hadoop集群配置主机masterhttp://i.hddly.cn/media/aTj3yJ6zTb.mp4http://i.hddly.cn/media/rCYroeQ4Oy.mp4克隆从机slavehttp://i.hddly.cn/media/krf7IlnGXo.mp4配置SSH免密码登录http://i.hddly.cn/media/3uGG8pOopw.mp4复制文件和配置时间同步服务http://i.hddly.cn/media/93HBaY5MyQ.mp4启动关闭监控集群http://i.hddly.cn/media/DH440uBnLH.mp4任务2.1安装及配置虚拟机软件准备centos7https://mirrors.aliyun.com/centos/7.9.20 ...crt工具http://i.hddly.cn/b37066/file/crt.rar或 \\10.255.10.50\public\crt或 超星->Hadoop->资料->tools->securecrt32.rarvmwarehttp://i.hddly.cn/b37066/file/wmware10.r ...Hadoop(建好虚拟机后在linux下载)wget https://mirrors.aliyun.com/apache/h ...或wget http://10.255.10.50/file/hadoop-3. ...官方资源Hadoop官网https://hadoop.apache.org/安装文档https://hadoop.apache.org/docs/stable/ha ...Hadoop的APIhttps://hadoop.apache.org/docs/stable/ap ...创建Linux虚拟机使用WMWare Workstation或者其它方式创建的虚拟机设置固定IPvi /etc/sysconfig/network-scripts/ifcfg- ...ifcfg-ens33在不同机器上可能不同,有可能是ifcfg-eth0IPADDR根据本机网络修改GATEWAY根据本机网络修改DNS1根据本机网络修改远程连接虚拟机crt工具更改hostname修改显示主机名hostnamectl set-hostname master修改网络主机名vi /etc/sysconfig/networkHOSTNAME=master使用主机名生效systemctl restart network检查hostnamehostname更新hosts说明:替换下面这些IP和名字后在命令行执行这些echo命令echo '' > /etc/hosts echo '10.255.10.65 ...关闭防火墙systemctl status firewalldsystemctl stop firewalldsystemctl disable firewalld.service安装NTPyum安装 ntpyum -y install ntp检查ntp服务systemctl status ntpd任务2.2安装JavaLinux下安装Java安装java的jdk方式一：通过yum安装更新reporm -f /etc/yum.repos.d/CentOS-Base.repowget -O /etc/yum.repos.d/CentOS-Base.rep ...或者: wget -O /etc/yum.repos.d/CentOS-Base ...yum查找jdkyum search java-1.8.0-openjdkyum安装jdkyum install java-1.8.0-openjdk方式二：通过scp复制jdkssh 10.255.10.31 --如果有提示密码，输入：root scp - ...查jdk路径java -verbose预期结果最后一行可能包含如下路径: /usr/lib/jvm/java-1.8. ...检查jdkjava -version预期结果: [root@c31 yum.repos.d]# java -vers ...安装jps方式一：通过yum安装yum安装jpsyum install -y java-1.8.0-openjdk-devel- ...方式二：通rpm安装sftp上传:\\10.255.10.50\public\soft\java\j ...cd \rootrpm -ivh jdk-8u321-linux-x64.rpm任务2.3搭建Hadoop完全分布式集群安装参考官方文档参考https://hadoop.apache.org/docs/stable/ha ...https://hadoop.apache.org/releases.htmlhttps://dlcdn.apache.org/hadoop/common/测试集群环境参考机器配置主从机都是4G内存2核,硬盘10G网络配置主机master10.255.10.31从机slave:10.255.10.3210.255.10.3310.255.10.34下载hadoop3.3.1创建子目录 hadoopcd /root mkdir hadoop cd hadoop在hadoop子目录中下载wget http://pan-yz.chaoxing.com/external ...wget https://dlcdn.apache.org/hadoop/com ...或 wget http://10.255.10.50/file/hadoop-3 ...或 http://i.hddly.cn/b37066/file/hadoop-3 ...或: mkdir /root/hadoop scp 10.255.10.31:/ ...安装hadoop3.3.1解压安装包cd /root/hadooptar -zxf hadoop-3.3.1.tar.gz -C /usr/loc ...说明:该解压过程可能需要较长时间,耐心等待在/usr/local/hadoop-3.3.1/etc/hadoop目录下修改 ...cd /usr/local/hadoop-3.3.1/etc/hadoopvi ./core-site.xml<configuration> <property> <name ...注意master要改为本集群的master名vi ./hadoop-env.sh说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考ja ...export JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...export HADOOP_HOME=/usr/local/hadoop-3.3 ...vi ./hdfs-site.xml<configuration> <property> <name>dfs ...注意master要改为本集群的master名vi ./mapred-site.xml<configuration> <property> <name>map ...注意master要改为本集群的master名vi ./yarn-site.xml<configuration> <!-- Site specific YARN ...注意master要改为本集群的master名vi ./yarn-env.sh说明:在文件上方空白行处添加如下：export JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...vi ./workers说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如 ...c32 c33 c34vi ./hadoop-env.sh说明:在刚才添加的export行后添加如下四行内容:HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...在//usr/local/hadoop-3.3.1/sbin目录下修改shell ...cd /usr/local/hadoop-3.3.1/sbinvi ./start-dfs.sh说明:在文件上方空白行处添加如下：HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...vi ./stop-dfs.sh说明:在文件上方空白行处添加如下：HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...vi ./start-yarn.sh说明:在文件上方空白行处添加如下：YARN_RESOURCEMANAGER_USER=root HADOOP_SE ...vi ./stop-yarn.sh说明:在文件上方空白行处添加如下：YARN_RESOURCEMANAGER_USER=root HADOOP_SE ...修改linux系统配置文件profile并使生效vi /etc/profile说明:在文件最下方export行后添加如下：export JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...source /etc/profile验证hadoop相关配置是否生效hadoop version说明：如果显示如下内容说明hadoop配置生效 [root@c31 ~]# ha ...配置SSH免密码登录进入root目录cd /root1,使用 ssh-keygen产生公钥与私钥对ssh-keygen -t rsa说明:运行过程中持续回车直到结束2,ssh-copy-id将公钥复制到远程机器中说明:将下方的master改为本群组的主机名,从机名,依次执行,复制过程有提示分 ...ssh-copy-id -i /root/.ssh/id_rsa.pub mas ...ssh-copy-id -i /root/.ssh/id_rsa.pub sla ...ssh-copy-id -i /root/.ssh/id_rsa.pub sl ...3,在master下分别输入 ssh slave1 验证是否成功登录说明:,如果过程没有提示密码,直接进入slave1,则成功,然后exit退回[root@c78 ~]# ssh c85 Last login: Mon Ma ...[root@c85 ~]# exit logout Connection to ...克隆虚拟机复制文件(所有从机)说明:针对从机还没有配置的情况才需要复制如下内容,以c32为例scp -r /usr/lib/jvm c32:/usr/lib/ scp -r ...在从机上生效 /etc/profile说明:使用crt切换到从机,然后执行以下命令source /etc/profile配置时间同步服务参考配置:http://home.hddly.cn:8000/article/2022/4 ...在主机master上操作vi /etc/ntp.confdriftfile /var/lib/ntp/drift restrict de ...service ntpd start & chkconfig ntpd on在所有的slave从机上操作vi /etc/ntp.conf说明:,master是本集群的主机名driftfile /var/lib/ntp/drift server mast ...systemctl stop ntpdntpdate masterservice ntpd start & chkconfig ntpd on启动关闭集群格式化主机的NameNodecd /usr/local/hadoop-3.3.1/bin/ ./hdfs ...启动集群命令说明:启动命令只需要在master上执行cd /usr/local/hadoop-3.3.1/sbin/./start-dfs.sh ./start-yarn.sh ./mr-jobhistory-daemon.sh start historys ...说明:使用jps查看若有以下进程说明启动正常11571 JobHistoryServer停止集群命令说明:启动命令只需要在master上执行cd /usr/local/hadoop-3.3.1/sbin/./stop-yarn.sh ./stop-dfs.sh ./mr-jobhistory-daemon.sh stop historyse ...启动节点命令master节点说明:启动节点命令可以在master上执行cd /usr/local/hadoop-3.3.1/sbin/hadoop-daemon.sh start namenodeyarn-daemon.sh start resourcemanagerslave节点说明:启动节点命令可以在slave上执行cd /usr/local/hadoop-3.3.1/sbin/./hadoop-daemon.sh start datanode./yarn-daemon.sh start nodemanager监控集群操作终端机hosts修改说明: 在windows文件中找到System32-->drivers-->et ...192.168.137.134 slave1 192.168.137.135 s ...HDFS监控hadoop3.x:http://192.168.137.133:9870/http://192.168.137.134:9864/datanode.htm ...http://192.168.137.135:9864/datanode.htm ...YARN监控hadoop3.x:http://192.168.137.133:8088/cl ...http://192.168.137.134:8042/nodehttp://192.168.137.135:8042/node日志监控hadoop3.x:http://192.168.137.133:19888/j ...其它参考在虚拟机上安装Hadoop2.x安装参考官方文档参考https://hadoop.apache.org/docs/stable/ha ...https://hadoop.apache.org/releases.htmlhttps://dlcdn.apache.org/hadoop/common/更新配置参考http://i.hddly.cn/b37066/lab/p2/Centos%2 ...测试集群环境参考机器配置主从机都是4G内存2核,硬盘10G网络配置主机master10.255.10.65从机slave:10.255.10.22210.255.10.21410.255.10.175安装jdk1.7yum install java-1.7.0-openjdk java -verbose/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.26 ...安装jpsyum install -y java-1.7.0-openjdk-devel. ...修改hosts说明:替换下面这些IP和名字后在命令行执行这些echo命令echo '' > /etc/hosts echo '10.255.10.65 ...下载Hadoop v2.10.1创建子目录 hadoopcd /root mkdir hadoop cd hadoop在hadoop子目录中下载wget https://dlcdn.apache.org/hadoop/com ...或 wget http://10.255.10.50/file/hadoop-2 ...安装 Hadoop2.10.1解压安装包cd /root/hadooptar -zxf hadoop-2.10.1.tar.gz -C /usr/lo ...说明:该解压过程可能需要较长时间,耐心等待在/usr/local/hadoop-2.10.1/etc/hadoop目录下修 ...cd /usr/local/hadoop-2.10.1/etc/hadoopvi ./core-site.xml<configuration> <property> <name ...注意master要改为本集群的master名vi ./hadoop-env.sh说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考ja ...export JAVA_HOME=/usr/lib/jvm/java-1.7.0 ...export HADOOP_HOME=/usr/local/hadoop-2.1 ...vi ./hdfs-site.xml<configuration> <property> <name>dfs ...注意master要改为本集群的master名vi ./mapred-site.xml<configuration> <property> <name>map ...注意master要改为本集群的master名vi ./yarn-site.xml<configuration> <!-- Site specific YARN ...注意master要改为本集群的master名vi ./yarn-env.sh说明:在文件上方空白行处添加如下：export JAVA_HOME=/usr/lib/jvm/java-1.7.0 ...vi ./slaves说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如 ...c175 c214 c222vi ./hadoop-env.sh说明:在刚才添加的export行后添加如下四行内容:HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...在//usr/local/hadoop-2.10.1/sbin目录下修改shel ...cd /usr/local/hadoop-2.10.1/sbinvi ./start-dfs.sh说明:在文件上方空白行处添加如下：HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...vi ./stop-dfs.sh说明:在文件上方空白行处添加如下：HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...vi ./start-yarn.sh说明:在文件上方空白行处添加如下：YARN_RESOURCEMANAGER_USER=root HADOOP_SE ...vi ./stop-yarn.sh说明:在文件上方空白行处添加如下：YARN_RESOURCEMANAGER_USER=root HADOOP_SE ...修改linux系统配置文件profile并使生效vi /etc/profile说明:在文件最下方export行后添加如下：export JAVA_HOME=/usr/lib/jvm/java-1.7.0 ...source /etc/profile验证hadoop相关配置是否生效hadoop version说明：如果显示如下内容说明hadoop配置生效 [root@c65 sbin]# ...配置SSH免密码登录进入root目录cd /root1,使用 ssh-keygen产生公钥与私钥对ssh-keygen -t rsa说明:运行过程中持续回车直到结束2,ssh-copy-id将公钥复制到远程机器中说明:将下方的master改为本群组的主机名,从机名,依次执行,复制过程有提示分 ...ssh-copy-id -i /root/.ssh/id_rsa.pub mas ...ssh-copy-id -i /root/.ssh/id_rsa.pub sla ...ssh-copy-id -i /root/.ssh/id_rsa.pub sl ...3,在master下分别输入 ssh slave1 验证是否成功登录说明:,如果过程没有提示密码,直接进入slave1,则成功,然后exit退回[root@c78 ~]# ssh c85 Last login: Mon Ma ...[root@c85 ~]# exit logout Connection to ...克隆虚拟机复制文件(所有从机)说明:针对从机还没有配置的情况才需要复制如下内容,以目标机器c175为例scp /etc/hosts c175:/etc/hosts scp -r /u ...在从机上生效 /etc/profile说明:使用crt切换到从机,然后执行以下命令source /etc/profile配置时间同步服务参考配置:http://home.hddly.cn:8000/article/2022/4 ...在主机master上操作vi /etc/ntp.confdriftfile /var/lib/ntp/drift restrict de ...service ntpd start & chkconfig ntpd on在所有的slave从机上操作vi /etc/ntp.conf说明:,master是本集群的主机名driftfile /var/lib/ntp/drift server mast ...systemctl stop ntpdntpdate masterservice ntpd start & chkconfig ntpd on启动关闭集群格式化主机的NameNodecd /usr/local/hadoop-2.10.1/bin/ ./hdfs ...启动集群命令说明:启动命令只需要在master上执行cd /usr/local/hadoop-2.10.1/sbin/./start-dfs.sh ./start-yarn.sh ./mr-jobhistory-daemon.sh start historys ...说明:使用jps查看若有以下进程说明启动正常11571 JobHistoryServer停止集群命令说明:启动命令只需要在master上执行cd /usr/local/hadoop-2.10.1/sbin/./stop-yarn.sh ./stop-dfs.sh ./mr-jobhistory-daemon.sh stop historyse ...启动节点命令master节点说明:启动节点命令可以在master上执行cd /usr/local/hadoop-2.10.1/sbin/./hadoop-daemon.sh start namenode./yarn-daemon.sh start resourcemanagerslave节点说明:启动节点命令可以在slave上执行,可在从机配置修改后或者重启后执行cd /usr/local/hadoop-2.10.1/sbin/./hadoop-daemon.sh start datanode或: hdfs --daemon start./yarn-daemon.sh start nodemanager监控集群操作终端机hosts修改说明: 在windows文件中找到System32-->drivers-->et ...10.255.10.222 c222 10.255.10.214 c214 10 ...HDFS监控http://c65:50070/http://c175:50075/datanode.htmlhttp://c214:50075/datanode.htmlYARN监控http://c65:8088/http://c175:8042/nodehttp://c214:8042/node日志监控http://c65:19888/在笔记本VM上安装Hadoop3.x集群环境笔记本配置内存,16G硬盘,500GCPU,4核8处理器,2.1GHz虚拟机配置主机master内存:4G处理器:分配4处理器,每处理器2核硬盘SATA:48G网络适配器: NAT从机slave内存:2G处理器: 分配2处理器,每处理器2核硬盘SATA:48G网络适配器:NAT网络配置主机master192.168.137.100从机slave:192.168.137.101192.168.137.102192.168.137.103安装及配置虚拟机软件准备centos7https://mirrors.aliyun.com/centos/7.9.20 ...crt工具http://i.hddly.cn/b37066/file/crt.rarvmwarehttp://i.hddly.cn/b37066/file/wmware10.r ...Hadoop(建好虚拟机后在linux下载)wget https://mirrors.aliyun.com/apache/h ...wmware安装与Linux安装安装WMWare Workstation10，注册配置wmware的虚拟网络编辑->虚拟网络编辑器添加网络->选择Nat模式->NAT设置网关:192.168.137.2选中:将主机虚拟适配器连接到此网络 选中:使用本地DHCP服务将IP地址分配给虚 ...子网IP:192.168.137.0,子网掩码:255.255.255.0在WMWare上创建虚拟机创建master主机选择默认的:典型下一步，选择安装程序光盘映像文件->浏览->前面下载的文件: CentOS-7- ...下一步，选中Linux操作系统，CentOS64位下一步，设置虚拟机名称:CentOS_Hadoop3.x_master,选择位置下一步，最大磁盘48G，将虚拟磁盘存储为单个文件完成设置，开始安装配置master网络右击从机master->设置内存:2G处理器: 分配2处理器,每处理器2核硬盘SATA:48G网络适配器:NAT开机master安装linux选择en设置安装源设置目标disk设置root用户密码为root开机master进入linux设置固定IPvi /etc/sysconfig/network-scripts/ifcfg- ...关闭防火墙systemctl stop firewalldsystemctl disable firewalld.service配置SSHrpm -qa | grep ssh如果没有sshyum install opensshservice sshd startchkconfig --level 5 sshd onsystemctl status sshd如果没有启动sshdyum -y install openssh openssh-clients o ...yum install -y openssh-server openssh-cl ...yum install initscripts -ychkconfig --add sshdservice sshd start远程连接虚拟机crt工具crt工具安装crt工具的配置与连接虚拟机以下linux操作如无说明均在crt里操作更改hostname修改显示主机名hostnamectl set-hostname master修改网络主机名vi /etc/sysconfig/networkHOSTNAME=mastersystemctl restart network更新hosts说明:替换下面这些IP和名字后在命令行执行这些echo命令echo '' > /etc/hosts echo '192.168.137.1 ...安装NTPyum -y install ntp虚拟机安装Java更新repoyum -y install wgetmv /etc/yum.repos.d/CentOS-Base.repo /e ...wget -O /etc/yum.repos.d/CentOS-Base.rep ...yum安装jdkyum install java-1.8.0-openjdk检查jdkjava -version预期结果: [root@c31 yum.repos.d]# java -vers ...查jdk路径java -verbose预期结果最后一行可能包含如下路径: /usr/lib/jvm/java-1.8. ...yum安装jpsyum install -y java-1.8.0-openjdk-devel. ...虚拟机安装Hadoop集群配置主机master下载安装hadoop3.3.1mkdir /root/hadoop cd /root/hadoop wget ...配置主机hadoop3.3.1在/usr/local/hadoop-3.3.1/etc/hadoop目录下修改 ...cd /usr/local/hadoop-3.3.1/etc/hadoopvi ./core-site.xml<configuration> <property> <name ...vi ./hadoop-env.sh说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考ja ...export JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...export HADOOP_HOME=/usr/local/hadoop-3.3 ...vi ./hdfs-site.xml<configuration> <property> <name>dfs ...vi ./mapred-site.xml<configuration> <property> <name>map ...vi ./yarn-site.xml<configuration> <!-- Site specific YARN ...vi ./yarn-env.sh说明:在文件上方空白行处添加如下：export JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...vi ./workers说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如 ...slave1 slave2 slave3vi ./hadoop-env.sh说明:在刚才添加的export行后添加如下5行内容:HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...在//usr/local/hadoop-3.3.1/sbin目录下修改shell ...cd /usr/local/hadoop-3.3.1/sbinvi ./start-dfs.sh说明:在文件上方空白行处添加如下：HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...vi ./stop-dfs.sh说明:在文件上方空白行处添加如下：HDFS_DATANODE_USER=root HDFS_DATANODE_SE ...vi ./start-yarn.sh说明:在文件上方空白行处添加如下：YARN_RESOURCEMANAGER_USER=root HADOOP_SE ...vi ./stop-yarn.sh说明:在文件上方空白行处添加如下：YARN_RESOURCEMANAGER_USER=root HADOOP_SE ...修改linux系统配置文件profile并使生效vi /etc/profile说明:在文件最下方export行后添加如下：export JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...source /etc/profile验证hadoop相关配置是否生效hadoop version说明：如果显示如下内容说明hadoop配置生效 [root@c31 ~]# ha ...克隆从机slave从master克隆出slave主机master关机快照 wmware左侧树图->右击主机master->管理->克隆->虚拟机当前状态- ...克隆出slave1右击从机slave1->设置内存:2G处理器: 分配2处理器,每处理器2核硬盘SATA:48G网络适配器:NAT开机slave1进入linux设置固定IPvi /etc/sysconfig/network-scripts/ifcfg- ...hostname修改hostnamectl set-hostname slave1vi /etc/sysconfig/networkHOSTNAME=slave1克隆出slave2右击从机slave2->设置内存:2G处理器: 分配2处理器,每处理器2核硬盘SATA:48G网络适配器:NAT开机slave2进入linux设置固定IPvi /etc/sysconfig/network-scripts/ifcfg- ...hostname修改hostnamectl set-hostname slave2vi /etc/sysconfig/networkHOSTNAME=slave2克隆出slave3右击从机slave3->设置内存:2G处理器: 分配2处理器,每处理器2核硬盘SATA:48G网络适配器:NAT开机slave3进入linux设置固定IPvi /etc/sysconfig/network-scripts/ifcfg- ...hostname修改hostnamectl set-hostname slave3vi /etc/sysconfig/networkHOSTNAME=slave3配置SSH免密码登录说明:在主机master上执行进入root目录cd /root1,使用 ssh-keygen产生公钥与私钥对说明:运行过程中持续回车直到结束ssh-keygen -t rsa2,ssh-copy-id将公钥复制到远程机器中说明:将下方的master改为本群组的主机名,从机名,依次执行,复制过程有提示分 ...ssh-copy-id -i /root/.ssh/id_rsa.pub mas ...ssh-copy-id -i /root/.ssh/id_rsa.pub sla ...ssh-copy-id -i /root/.ssh/id_rsa.pub sl ...ssh-copy-id -i /root/.ssh/id_rsa.pub sl ...3,通过ssh slave 验证是否成功免密登录从机说明:,如果过程没有提示密码,直接进入slave1,则成功,然后exit退回到m ...ssh slave1 exitssh slave2 exitssh slave3 exit复制文件到从机说明:在主机master上执行scp -r /usr/lib/jvm slave1:/usr/lib/ scp ...scp -r /usr/lib/jvm slave2:/usr/lib/ scp ...scp -r /usr/lib/jvm slave3:/usr/lib/ scp ...在从机上生效 /etc/profile说明:使用ssh切换到从机,然后执行source命令,然后exit退回到mast ...ssh slave1 source /etc/profile exitssh slave2 source /etc/profile exitssh slave3 source /etc/profile exit配置时间同步服务参考配置:http://home.hddly.cn:8000/article/2022/4 ...在主机master上操作vi /etc/ntp.confdriftfile /var/lib/ntp/drift restrict de ...service ntpd start & chkconfig ntpd on在所有的slave从机上操作vi /etc/ntp.conf说明:,master是本集群的主机名driftfile /var/lib/ntp/drift server mast ...systemctl stop ntpdntpdate masterservice ntpd start & chkconfig ntpd on启动关闭集群说明:格式和启动停止集群命令只需要在master上执行格式化主机的NameNode/usr/local/hadoop-3.3.1/bin/hdfs namenod ...启动集群命令cd /usr/local/hadoop-3.3.1/sbin/./start-dfs.sh ./start-yarn.sh mapred --daemon start historyserverjps说明:使用jps查看若有Namenode,SecondaryNameNode,R ...停止集群命令cd /usr/local/hadoop-3.3.1/sbin/./stop-yarn.sh ./stop-dfs.sh mapred --daemon stop historyserver启动节点命令master节点说明:启动节点命令可以在master上执行cd /usr/local/hadoop-3.3.1/sbin/hadoop-daemon.sh start namenodeyarn-daemon.sh start resourcemanagerslave节点说明:启动节点命令可以在slave上执行cd /usr/local/hadoop-3.3.1/sbin/./hadoop-daemon.sh start datanode./yarn-daemon.sh start nodemanager监控集群操作终端机hosts修改说明: 在windows文件中找到System32-->drivers-->et ...192.168.137.100 master 192.168.137.101 s ...HDFS监控:http://master:9870/http://slave1:9864/datanode.htmlhttp://slave2:9864/datanode.htmlYARN监控http://master:8088/clusterhttp://slave1:8042/nodehttp://slave2:8042/node日志监控http://master:19888/jobhistory在云服务器搭建完全分布式hadoop问题： 在局域网内配置hadoop，只要在/etc/hosts中配置各个节点的局 ...解决方案：在master节点vim /etc/hostsmaster的IP用内网ipslave1的IP用外网ip在slave1节点vim /etc/hostsmaster的IP用外网ipslave1的IP用内网ip端口配置参考Hadoop3.xcore-default.xmlhdfs://master:9864hdfs-default.xmldfs.namenode.secondary.http-address ...dfs.namenode.secondary.https-address ...dfs.datanode.address 0.0.0.0:9866 ...dfs.datanode.http.address 0.0.0.0 ...dfs.datanode.ipc.address 0.0.0.0: ...dfs.namenode.http-address 0.0.0.0 ...dfs.datanode.https.address 0.0.0. ...dfs.namenode.https-address 0.0.0. ...dfs.namenode.backup.address 0.0.0 ...dfs.namenode.backup.http-address ...dfs.journalnode.rpc-address 0.0.0 ...dfs.journalnode.http-address 0.0. ...dfs.journalnode.https-address 0.0 ...dfs.balancer.address 0.0.0.0:0dfs.mover.address 0.0.0.0:0dfs.storage.policy.satisfier.address ...mapred-default.xmlmapreduce.jobhistory.address 0.0. ...mapreduce.jobhistory.webapp.address ...mapreduce.jobhistory.webapp.https.addres ...mapreduce.jobhistory.admin.address ...yarn-default.xmlyarn.nodemanager.webapp.https.address ...yarn.sharedcache.admin.address 0. ...yarn.sharedcache.webapp.address 0 ...yarn.sharedcache.uploader.server.address ...yarn.sharedcache.client-server.address ...yarn.nodemanager.amrmproxy.address ...yarn.router.webapp.address 0.0.0. ...yarn.router.webapp.https.address ...Hadoop 3 端口号(2.x->3.x)的改变Namenode 端口NNPorts Namenode 8020 → 9820NNPorts NN HTTP UI 50070 → 9870NNPorts NN HTTPS UI 50470 → 9871Secondary NN 端口SNN ports SNN HTTP 50091 → 9869SNN ports SNN HTTP UI 50090 → 9868Datanode 端口:DN ports DN IPC 50020 → 9867DN ports DN 50010 → 9866DN ports DN HTTP UI 50075 → 9864DN ports Namenode 50475 → 9865Hadoop2.x修改配置文件更改hostname修改主机名： 例如：master 、slave1、slave2 (1) 修改显示 ...映射IP地址编辑/etc/hosts 192.168.137.133 master mast ...新增slave3 192.168.137.136 slave3 slave3.c ...配置用户权限11.5 /hadoop-3.3.0/sbin下修改 start-dfs.sh、 ...11.6 修改 start-yarn.sh、stop-yarn.sh 文件： 修 ...配置workers批量启动脚本11.8 配置workers批量启动脚本： 从name node 服务器上配置批 ...Hadoop与Java配套Jdk1.7->Hadoop2.0Jdk1.8->Hadoop3.0关闭防火墙相关命令(仅供参考)（1）设置开机启用防火墙：systemctl enable firewalld. ...（2）设置开机禁用防火墙：systemctl disable firewalld ...（3）启动防火墙：systemctl start firewalld（4）关闭防火墙：systemctl stop firewalld（5）检查防火墙状态：systemctl status firewalld从机Slave迁移到另一个集群1，修改/etc/host,将master的IP切换到新的集群master的IP2,备份原来集群的的VERSIONcd /data/hadoop/hdfs/data/currentcp ./VERSION ./VERSION_192.168.31.21 2.查找新的集群的IDcd /data/hadoop/hdfs/data/currentmore ./VERSION找到:clusterID=CID-4e682a36-d31c-41ce-a749 ...3. 修改VERSION ，将新的集群的ID替换旧的4.重启集群实训实训1为Hadoop集群新增节点实训目的掌握Hadoop集群的掌握JDK的安装方法实训要点掌握Hadoop集群的搭建掌握Hadoop集群的启动与关闭熟悉Hadoop集群的监控实训需求说明在已有的集群中新增一个子节点实现思路及步骤克隆虚拟机参考其它从节点修改新增节点的配置在主节点和所有子节点修改workder,添加新增节点名称配置ssh免密登录到新增节点重启集群，或单独启动新增节点具体操作参考新增云节点1,在平台新增虚拟机10.255.10.311,改名:C312,进入linux->改/etc/sysconfig/network-scrip ...3,保存:wq!4,重启服务 systemctl restart network2,更改hostname1,修改显示主机名：hostnamectl set-hostname C312,修改网络主机名：vi /etc/sysconfig/networkHOSTNAME=C313,重启network服务systemctl restart network新增全新节点1,clone虚拟机centos72,设置IP地址： vi /etc/sysconfig/network-scri ...3,在虚拟机属性中设置网络类型为NAT4,更改hostname1,修改显示主机名：hostnamectl set-hostname slave ...2,修改网络主机名：vi /etc/sysconfig/networkHOSTNAME=slave35,修改hosts编辑所有的节点 vi /etc/hosts 192.168.137.133 ma ...crt中批量添加sed -i '1i\192.168.137.137 slave4 slave4 ...表示向/etc/hosts文件里的第1行的前面添加内容6,从master复制公钥到slave3ssh-copy-id -i /root/.ssh/id_rsa.pub sl ...7,复制文件拷贝hadoop安装文件到集群slave节点: scp -r /usr/loca ...8,配置时间同步ntpyum -y install ntp修改/etc/ntp.conf文件，注释掉server开头的行，并添加:serv ...service iptables stop & chkconfig iptabl ...ntpdate masterservice ntpd start & chkconfig ntpd on9,安装javayum卸载jdkyum remove java-1.8.0-openjdkyum remove -y java-1.8.0-openjdk-devel.x ...yum安装jdkyum install java-1.8.0-openjdkPackage java-1.8.0-openjdk-devel.x86_64 ...yum安装jpsyum install -y java-1.8.0-openjdk-devel. ...copy目录cp -r ./jre-1.8.0-openjdk-1.8.0.322.b06- ...cp -r /usr/lib/jvm/jre /usr/lib/jvm/jav ...10,修改/etc/profileexport JAVA_HOME=/usr/lib/jvm/java-1.8.0 ...11,关闭防火墙systemctl disable firewalldservice stop firewalld12,启动子节点　　hadoop-daemon.sh start datanode　　yarn-daemon.sh start nodemanager新增CLone节点1,clone虚拟机slave12,设置IP地址： vi /etc/sysconfig/network-scri ...4,更改hostname1,修改显示主机名：hostnamectl set-hostname slave ...2,修改网络主机名：vi /etc/sysconfig/networkHOSTNAME=slave35,修改hosts&workercrt中批量添加hostsed -i '1i\192.168.137.137 slave4 slave4 ...表示向/etc/hosts文件里的第1行的前面添加内容crt中添加workersed -i '1i\slave4' /usr/local/hadoop-3. ...6,从master复制公钥到slave3ssh-copy-id -i /root/.ssh/id_rsa.pub sl ...7,删除数据文件 rm -rf /data/hadoop/hdfs/data/c ...可以删除/data/hadoop/hdfs整个目录12,启动子节点　　hadoop-daemon.sh start datanode　　yarn-daemon.sh start nodemanager ./start-balancer.sh -threshold 3集群自均衡实训作业要求1，环境说明:本小组主机:,本小组成员机:,本成员机:,新增从机:2，简要说明新增从机的步骤，并截取过程截图3，截图：新增从机后hdfs:http://master:9870的node列表 ...4，截图：新增从机后yarn:http://master:8088的node列表 ...实训2实现集群节点间的时间同步训练要点掌握NTP服务的启动与关闭，参考 blog：http://home.hddly. ...掌握集群同步时间命令需求说明在master上或slave上配置/etc/ntp.conf停启ntp服务观察ntp服务实现步骤修改/etc/ntp.conf配置文件关闭ntp服务：systemctl stop ntpd同步时间: ntpdate server启动ntp服务: systemctl enable ntpd systemctl ...观察ntp服务: ntpstat ntpq -p作业要求1，环境说明:本小组主机:,本小组成员机:,本成员机:2，拍照截取本小组集群中本成员虚拟机上/etc/ntp.conf内容3，截图：ntpstat的结果4，截图：ntpq -p的结果实训3编写shell脚本实现配置文件的分发训练要点掌握集群中配置文件或包文件的批量分发 熟练使用scp命令熟练使用shell脚本需求说明在master上的hadoop配置目录下创建./scp.sh编辑scp.sh，添加scp命令，复制本目录下所有文件到从机运行./scp.sh，观察配置文件是否成功复制到从机实现步骤在master上: cd /usr/local/hadoop-3.3.1/etc ...测试scp命令,复制配置文件到10.255.10.123的/root/mynam ...参考下方脚本 [root@master hadoop]# vi ./scp.sh ...chmod 777 ./scp.sh运行./scp.sh,观察运行情况作业要求1，环境说明:本小组主机:,本小组成员机:,本成员机:2，拍照截取scp * 10.255.10.123:/root/myname/ ...3，拍照截取./scp.sh的脚本，以及运行结果实训4优化配置文件分发脚本chmod 777 /usr/local/hadoop-3.3.1/etc/ha ...[root@master ~]# more ./scp_workers.sh ...实训5实现从节点的服务重启vi /root/restart_master.shvi /root/restart_master.sh /usr/local/ha ...常见问题添加节点时在ssh-copy-id报异常命令:ssh-copy-id -i /root/.ssh/id_rsa.pub ...现象:Host key verification failed. 处理:vi /root/.ssh/known_hosts ,将slave3这行 ...添加节点后,节点未能加入成功日志:ERROR org.apache.hadoop.hdfs.StateCha ...日志:org.apache.hadoop.hdfs.protocol.Unreg ...处理:删除新增节点的数据目录:rm -rf /data/hadoop/hdfs/ ...ERROR: Attempting to operate on hdfs nam ...Starting namenodes on [master] ERROR: At ...在/hadoop/sbin路径下： 将start-dfs.sh，stop-dfs ...master: ERROR: JAVA_HOME is not set and ...vi /usr/local/hadoop-3.3.1/etc/hadoop/ha ...ERROR: but there is no YARN_RESOURCEMANA ...还有，start-yarn.sh，stop-yarn.sh顶部也需添加以下： ...master机NameNode没有启动先手动启动namenode后，再启动dfs./hadoop-daemon.sh start namenode ./hado ..../start-dfs.sh slave机NodeManager没有启动配置文件core-site.xml和hdfs-site.xml中端口hdfs监控：http://192.168.137.133:9870/Configured Capacity:0 B删除 /etc/hosts 文件中127.0.0.1行正确的 [root@master sbin]# more /etc/hosts ...http://slave1:9864无法打开 在windows文件中找到System32-->drivers-->etc,进入 ...192.168.137.134 slave1 192.168.137.135 s ...yarn监控：http://192.168.137.133:8088/http://slave1:8042无法打开在windows文件中找到System32-->drivers-->etc,进入 ...若权限问题,打开属性,添加用户192.168.137.134 slave1 192.168.137.135 s ...启动报jdk错误ERROR: JAVA_HOME /usr/lib/jvm/java-1.8.0 ...java -verbose查看/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.32 ...报jps命令不存在现象[root@c78 ~]# jps-bash: jps: command not found处理先卸载jpsyum remove -y java-1.8.0-openjdk-devel.x ...再安装jpsyum install -y java-1.8.0-openjdk-devel. ...wmware启动时提示内部错误尝试一:计算机管理->服务->检查WMware5个服务是否都开启datanode移动到另一集群问题切换集群后,datanode日志报错java.io.IOException: Incompatible cluste ... ERROR org.apache.hadoop.hdfs.server.dat ...错误截图http://i.hddly.cn/media/SecureCRT_81gsBF ...迁移处理1,备份原来集群的的VERSIONcd /data/hadoop/hdfs/data/currentcp ./VERSION ./VERSION_192.168.31.21 2.查找新的集群的IDcd /data/hadoop/hdfs/data/currentmore ./VERSION找到:clusterID=CID-4e682a36-d31c-41ce-a749 ...3. 修改VERSION ，将新的集群的ID替换旧的4.重启集群新增节点在web监控界面没有增加datanode尝试清空浏览器数据，刷新页面新增节点在yarn监控界面没有增加节点分析没有增加进来的节点的日志异常yarn.client.DefaultNoHARMFailoverProxyPr ...处理:增加8031端口的转发由于master主机在NAT内网，因此在外网的机器要访问内网的master,需要 ...测试:在linux上测试example的Pi,可以创建任务,yarn web上也能看到nptd服务异常no server suitable for synchronization f ...尝试关闭防火墙systemctl disable firewalldsystemctl stop firewalld尝试:ntpdate time.nist.gov客户端尝试: ntpdate master hdfs dfsadmin -printTopology正常的显示没有网关[root@master sbin]# hdfs dfsadmin -print ...异常:显示的地址是网关地址[root@master logs]# hdfs dfsadmin -print ...更新network网卡uuid无效重装wm workstation 10无效升级wm workstation 16无效查找是否有类似:squidGuard的代理应用，有的话卸载试试hdfs missing blocks. The following files ...参考：http://www.manongjc.com/detail/28-noa ...解决方案查看丢失文件情况：hdfs fsck /删除丢失文件的数据hdfs fsck -delete刷新 hdfs web 界面恢复正常HDFS磁盘空间不足现象:在多次上传到Hdfs和下载文件到本地时出现no space情况处理:当前Linux至少分配50G空间,但实际使用为10G,可以扩大步骤1：创建分区vda3fdisk /dev/vda 进入后输入:n 然后选择主分区:p 使用默认分区号 ...partprobe使用分区更改立即生效步骤2：创建物理卷pvcreate /dev/vda3步骤3： 扩展卷组vgextend centos /dev/vda3步骤4：扩展逻辑卷lvextend -L +10G /dev/centos/rootxfs_growfs /dev/mapper/centos-root使用cp,替换当前时区为上海时区，如有提示，输入yes进行确认 cp /usr/ ...Liux时区设置使用cp,替换当前时区为上海时区，如有提示，输入yes进行确认 cp /usr/ ...HDFS重新格式化1,删除所有从机的/data/hadoop目录: rm -rf /data/ha ...2,删除hadoop目录下的/tmp目录 rm -rf /usr/local/h ...3,停止集群 stop-all.sh4,格式化hdfs hdfs namenode -format3,启动集群 start-all.shHDFS块修复hdfs debug recoverLease -path 文件绝对路径 -r ...hdfs fsck -delete 文件绝对路径检测缺失块hdfs fsck -list-corruptfileblockshdfs fsck / 查看某一个文件的情况hdfs fsck /路径/文件名 -locations -blocks -fi ...手动修复hdfs debug recoverLease -path 文件位置 -retr ...删除所有损坏的块hdfs fsck / -delete降低hdfs上权限给根目录加权限hdfs dfs -chmod -R 777 /退出安全模式查看hdfs safe modehadoop dfsadmin -safemode get退出hdfs safe modehadoop dfsadmin -safemode leave版本历史Ver1.1-20220121初始版本Ver1.2-20230301修改Hadoop部分操作步骤，增加常见问题(来自大数据综合实训的hadoop集群 ...
hide
第2章Hadoop集群搭建
hide
任务前言
hide
学习目标
leaf
（1）掌握安装配置虚拟机。
leaf
（2）掌握如何在本地和Linux虚拟机中安装JDK。
leaf
（3）掌握搭建Hadoop完全分布式集群环境。
hide
任务背景
hide
集群环境分类
hide
单机版
leaf
一台机器上运行Hadoop
hide
伪分布式
leaf
一台单机上模拟组建的多节点集群
hide
完全分布式
leaf
是在多台单机上组建的分布式集群
hide
任务指导
leaf
http://bigdata.hddly.cn/b37066/B37066L02.html
hide
学习视频
hide
在笔记本上安装hadoop3.x
hide
环境概述与环境准备
leaf
http://i.hddly.cn/media/G5dykd0RWL.mp4 User Link
hide
安装及创建虚拟机
leaf
http://i.hddly.cn/media/HUSTaLf3ok.mp4 User Link
leaf
http://i.hddly.cn/media/IrlUMiUIuu.mp4 User Link
hide
配置虚拟机
leaf
http://i.hddly.cn/media/MDQuzpBM4A.mp4 User Link
leaf
http://i.hddly.cn/media/rUvi0fQnOj.mp4 User Link
hide
虚拟机安装Java
leaf
http://i.hddly.cn/media/SiF5MekLLr.mp4 User Link
hide
虚拟机安装Hadoop集群
hide
配置主机master
leaf
http://i.hddly.cn/media/aTj3yJ6zTb.mp4 User Link
leaf
http://i.hddly.cn/media/rCYroeQ4Oy.mp4 User Link
hide
克隆从机slave
leaf
http://i.hddly.cn/media/krf7IlnGXo.mp4 User Link
hide
配置SSH免密码登录
leaf
http://i.hddly.cn/media/3uGG8pOopw.mp4 User Link
hide
复制文件和配置时间同步服务
leaf
http://i.hddly.cn/media/93HBaY5MyQ.mp4 User Link
hide
启动关闭监控集群
leaf
http://i.hddly.cn/media/DH440uBnLH.mp4 User Link
hide
任务2.1安装及配置虚拟机
hide
软件准备
hide
centos7
leaf
https://mirrors.aliyun.com/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-Minimal-2009.iso
hide
crt工具
leaf
http://i.hddly.cn/b37066/file/crt.rar
leaf
或 \\10.255.10.50\public\crt
leaf
或 超星->Hadoop->资料->tools->securecrt32.rar
hide
vmware
leaf
http://i.hddly.cn/b37066/file/wmware10.rar
hide
Hadoop(建好虚拟机后在linux下载)
leaf
wget https://mirrors.aliyun.com/apache/hadoop/core/hadoop-3.3.1/hadoop-3.3.1.tar.gz
leaf
或wget http://10.255.10.50/file/hadoop-3.3.1.tar.gz
hide
官方资源
hide
Hadoop官网
leaf
https://hadoop.apache.org/ User Link
hide
安装文档
leaf
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html
hide
Hadoop的API
leaf
https://hadoop.apache.org/docs/stable/api/index.html User Link
hide
创建Linux虚拟机
leaf
使用WMWare Workstation
leaf
或者其它方式创建的虚拟机
hide
设置固定IP
hide
vi /etc/sysconfig/network-scripts/ifcfg-ens33
IPADDR=192.168.137.133
GATEWAY=192.168.137.2
NETMASK=255.255.255.0
DNS1=192.168.137.2
leaf
ifcfg-ens33在不同机器上可能不同,有可能是ifcfg-eth0
leaf
IPADDR根据本机网络修改
leaf
GATEWAY根据本机网络修改
leaf
DNS1根据本机网络修改
hide
远程连接虚拟机
leaf
crt工具
hide
更改hostname
hide
修改显示主机名
leaf
hostnamectl set-hostname master
hide
修改网络主机名
hide
vi /etc/sysconfig/network
leaf
HOSTNAME=master
hide
使用主机名生效
leaf
systemctl restart network
hide
检查hostname
leaf
hostname
hide
更新hosts
leaf
说明:替换下面这些IP和名字后在命令行执行这些echo命令
leaf
echo '' > /etc/hosts
echo '10.255.10.65 c65 c65.centos.com' >> /etc/hosts
echo '10.255.10.175 c175 c175.centos.com' >> /etc/hosts
echo '10.255.10.214 c214 c214.centos.com' >> /etc/hosts
echo '10.255.10.222 c222 c222.centos.com' >> /etc/hosts
hide
关闭防火墙
leaf
systemctl status firewalld
leaf
systemctl stop firewalld
leaf
systemctl disable firewalld.service
hide
安装NTP
hide
yum安装 ntp
leaf
yum -y install ntp
hide
检查ntp服务
leaf
systemctl status ntpd
hide
任务2.2安装Java
hide
Linux下安装Java
hide
安装java的jdk
hide
方式一：通过yum安装
hide
更新repo
leaf
rm -f /etc/yum.repos.d/CentOS-Base.repo
leaf
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo User Link
leaf
或者: wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo
hide
yum查找jdk
leaf
yum search java-1.8.0-openjdk
hide
yum安装jdk
leaf
yum install java-1.8.0-openjdk
hide
方式二：通过scp复制jdk
leaf
ssh 10.255.10.31
--如果有提示密码，输入：root
scp -r /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64 10.255.10.145:/usr/lib/jvm/
--如果有提示yes/no,输入：yes
--如果有提示密码，输入：root
hide
查jdk路径
hide
java -verbose
leaf
预期结果最后一行可能包含如下路径:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
或
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
hide
检查jdk
hide
java -version
leaf
预期结果:
[root@c31 yum.repos.d]# java -version
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (build 25.322-b06, mixed mode)
[root@c31 yum.repos.d]#
hide
安装jps
hide
方式一：通过yum安装
hide
yum安装jps
leaf
yum install -y java-1.8.0-openjdk-devel-1.8.0.322.b06-1.el7_9.x86_64
hide
方式二：通rpm安装
Arrow Link
leaf
sftp上传:\\10.255.10.50\public\soft\java\jdk-8u321-linux-x64.rpm 到虚拟机的\root
leaf
cd \root
leaf
rpm -ivh jdk-8u321-linux-x64.rpm
hide
任务2.3搭建Hadoop完全分布式集群
Arrow Link
hide
安装参考
hide
官方文档参考
leaf
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html User Link
leaf
https://hadoop.apache.org/releases.html
leaf
https://dlcdn.apache.org/hadoop/common/
hide
测试集群环境参考
hide
机器配置
leaf
主从机都是4G内存2核,硬盘10G
hide
网络配置
hide
主机master
leaf
10.255.10.31
hide
从机slave:
leaf
10.255.10.32
leaf
10.255.10.33
leaf
10.255.10.34
hide
下载hadoop3.3.1
hide
创建子目录 hadoop
leaf
cd /root
mkdir hadoop
cd hadoop
hide
在hadoop子目录中下载
leaf
wget http://pan-yz.chaoxing.com/external/m/file/706265312586317824
leaf
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz
leaf
或 wget http://10.255.10.50/file/hadoop-3.3.1.tar.gz
leaf
或 http://i.hddly.cn/b37066/file/hadoop-3.3.1.tar.gz
leaf
或:
mkdir /root/hadoop
scp 10.255.10.31:/root/hadoop-3.3.1.tar.gz /root/hadoop/
hide
安装hadoop3.3.1
hide
解压安装包
leaf
cd /root/hadoop
hide
tar -zxf hadoop-3.3.1.tar.gz -C /usr/local
leaf
说明:该解压过程可能需要较长时间,耐心等待
hide
在/usr/local/hadoop-3.3.1/etc/hadoop目录下修改配置
leaf
cd /usr/local/hadoop-3.3.1/etc/hadoop
hide
vi ./core-site.xml
hide
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9864</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-3.3.1/tmp</value>
</property>
<property>
     <name>hadoop.native.lib</name>
     <value>false</value>
     <description>Should native hadoop libraries, if present, be used.
     </description>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./hadoop-env.sh
leaf
说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考java -verbose命令
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
leaf
export HADOOP_HOME=/usr/local/hadoop-3.3.1
hide
vi ./hdfs-site.xml
hide
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9868</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./mapred-site.xml
hide
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./yarn-site.xml
hide
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./yarn-env.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
hide
vi ./workers
leaf
说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如下：
leaf
c32
c33
c34
hide
vi ./hadoop-env.sh
leaf
说明:在刚才添加的export行后添加如下四行内容:
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
HADOOP_SHELL_EXECNAME=root
hide
在//usr/local/hadoop-3.3.1/sbin目录下修改shell脚本
leaf
cd /usr/local/hadoop-3.3.1/sbin
hide
vi ./start-dfs.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
hide
vi ./stop-dfs.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
hide
vi ./start-yarn.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hide
vi ./stop-yarn.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hide
修改linux系统配置文件profile并使生效
hide
vi /etc/profile
leaf
说明:在文件最下方export行后添加如下：
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
export HADOOP_HOME=/usr/local/hadoop-3.3.1
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin
Arrow Link
leaf
source /etc/profile
hide
验证hadoop相关配置是否生效
hide
hadoop version
leaf
说明：如果显示如下内容说明hadoop配置生效
[root@c31 ~]# hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /usr/local/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar
hide
配置SSH免密码登录
hide
进入root目录
leaf
cd /root
hide
1,使用 ssh-keygen产生公钥与私钥对
hide
ssh-keygen -t rsa
leaf
说明:运行过程中持续回车直到结束
hide
2,ssh-copy-id将公钥复制到远程机器中
leaf
说明:将下方的master改为本群组的主机名,从机名,依次执行,复制过程有提示分别输入:yes和root
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub master
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
hide
3,在master下分别输入 ssh slave1 验证是否成功登录
leaf
说明:,如果过程没有提示密码,直接进入slave1,则成功,然后exit退回
leaf
[root@c78 ~]# ssh c85
Last login: Mon Mar 7 19:50:05 2022 from 10.255.7.102
[root@c85 ~]#
Arrow Link
leaf
[root@c85 ~]# exit
logout
Connection to c85 closed.
[root@c78 ~]#
hide
克隆虚拟机
hide
复制文件(所有从机)
leaf
说明:针对从机还没有配置的情况才需要复制如下内容,以c32为例
leaf
scp -r /usr/lib/jvm c32:/usr/lib/
scp -r /usr/local/hadoop-3.3.1/ c32:/usr/local
scp /etc/profile c32:/etc/profile
hide
在从机上生效 /etc/profile
leaf
说明:使用crt切换到从机,然后执行以下命令
leaf
source /etc/profile
Arrow Link
hide
配置时间同步服务
hide
参考配置:
leaf
http://home.hddly.cn:8000/article/2022/4/13/32.html User Link
hide
在主机master上操作
Arrow Link Arrow Link
hide
vi /etc/ntp.conf
leaf
driftfile /var/lib/ntp/drift
restrict default nomodify
restrict 10.255.10.0 mask 255.255.255.0 nomodify
server cn.pool.ntp.org
server 127.127.1.0
fudge 127.127.1.0 stratum 10
leaf
service ntpd start & chkconfig ntpd on
hide
在所有的slave从机上操作
hide
vi /etc/ntp.conf
leaf
说明:,master是本集群的主机名
leaf
driftfile /var/lib/ntp/drift
server master
includefile /etc/ntp/crypto/pw
disable monitor
leaf
systemctl stop ntpd
leaf
ntpdate master
leaf
service ntpd start & chkconfig ntpd on
hide
启动关闭集群
hide
格式化主机的NameNode
leaf
cd /usr/local/hadoop-3.3.1/bin/
./hdfs namenode -format
hide
启动集群命令
leaf
说明:启动命令只需要在master上执行
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
./start-dfs.sh
leaf
./start-yarn.sh
hide
./mr-jobhistory-daemon.sh start historyserver
or
mapred --daemon start historyserver
leaf
说明:使用jps查看若有以下进程说明启动正常
leaf
11571 JobHistoryServer
hide
停止集群命令
leaf
说明:启动命令只需要在master上执行
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
./stop-yarn.sh
leaf
./stop-dfs.sh
leaf
./mr-jobhistory-daemon.sh stop historyserver
or
mapred --daemon stop historyserver
hide
启动节点命令
hide
master节点
leaf
说明:启动节点命令可以在master上执行
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
hadoop-daemon.sh start namenode
leaf
yarn-daemon.sh start resourcemanager
hide
slave节点
leaf
说明:启动节点命令可以在slave上执行
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
./hadoop-daemon.sh start datanode
leaf
./yarn-daemon.sh start nodemanager
hide
监控集群
hide
操作终端机hosts修改
leaf
说明:
在windows文件中找到System32-->drivers-->etc,进入到etc文件夹中就能看到hosts文件,添加如下行,其中IP和机器名要改为本小组集群的IP和名称
leaf
192.168.137.134 slave1
192.168.137.135 slave2
192.168.137.133 master
192.168.137.136 slave3
10.255.10.31 c31
10.255.10.32 c32
10.255.10.33 c33
10.255.10.34 c34
hide
HDFS监控
hide
hadoop3.x:http://192.168.137.133:9870/
leaf
http://192.168.137.134:9864/datanode.html
leaf
http://192.168.137.135:9864/datanode.html
hide
YARN监控
hide
hadoop3.x:http://192.168.137.133:8088/cluster
leaf
http://192.168.137.134:8042/node
leaf
http://192.168.137.135:8042/node
hide
日志监控
leaf
hadoop3.x:http://192.168.137.133:19888/jobhistory
hide
其它参考
hide
在虚拟机上安装Hadoop2.x
hide
安装参考
hide
官方文档参考
leaf
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html User Link
leaf
https://hadoop.apache.org/releases.html
leaf
https://dlcdn.apache.org/hadoop/common/
hide
更新配置参考
leaf
http://i.hddly.cn/b37066/lab/p2/Centos%206.8%E8%99%9A%E6%8B%9F%E6%9C%BA%E9%85%8D%E7%BD%AEHadoop2.6.txt User Link
hide
测试集群环境参考
hide
机器配置
leaf
主从机都是4G内存2核,硬盘10G
hide
网络配置
hide
主机master
leaf
10.255.10.65
hide
从机slave:
leaf
10.255.10.222
leaf
10.255.10.214
leaf
10.255.10.175
hide
安装jdk1.7
leaf
yum install java-1.7.0-openjdk
hide
java -verbose
leaf
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
hide
安装jps
leaf
yum install -y java-1.7.0-openjdk-devel.x86_64
hide
修改hosts
leaf
说明:替换下面这些IP和名字后在命令行执行这些echo命令
leaf
echo '' > /etc/hosts
echo '10.255.10.65 c65 c65.centos.com' >> /etc/hosts
echo '10.255.10.175 c175 c175.centos.com' >> /etc/hosts
echo '10.255.10.214 c214 c214.centos.com' >> /etc/hosts
echo '10.255.10.222 c222 c222.centos.com' >> /etc/hosts
hide
下载Hadoop v2.10.1
hide
创建子目录 hadoop
leaf
cd /root
mkdir hadoop
cd hadoop
hide
在hadoop子目录中下载
leaf
wget https://dlcdn.apache.org/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz
leaf
或 wget http://10.255.10.50/file/hadoop-2.10.1.tar.gz
hide
安装 Hadoop2.10.1
hide
解压安装包
leaf
cd /root/hadoop
hide
tar -zxf hadoop-2.10.1.tar.gz -C /usr/local
leaf
说明:该解压过程可能需要较长时间,耐心等待
hide
在/usr/local/hadoop-2.10.1/etc/hadoop目录下修改配置
leaf
cd /usr/local/hadoop-2.10.1/etc/hadoop
hide
vi ./core-site.xml
hide
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/log/hadoop/tmp</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./hadoop-env.sh
leaf
说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考java -verbose命令
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
leaf
export HADOOP_HOME=/usr/local/hadoop-2.10.1
hide
vi ./hdfs-site.xml
hide
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./mapred-site.xml
hide
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./yarn-site.xml
hide
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
</configuration>
leaf
注意master要改为本集群的master名
hide
vi ./yarn-env.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
hide
vi ./slaves
leaf
说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如下：
leaf
c175
c214
c222
hide
vi ./hadoop-env.sh
leaf
说明:在刚才添加的export行后添加如下四行内容:
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
HADOOP_SHELL_EXECNAME=root
hide
在//usr/local/hadoop-2.10.1/sbin目录下修改shell脚本
leaf
cd /usr/local/hadoop-2.10.1/sbin
hide
vi ./start-dfs.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
hide
vi ./stop-dfs.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
hide
vi ./start-yarn.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hide
vi ./stop-yarn.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hide
修改linux系统配置文件profile并使生效
hide
vi /etc/profile
leaf
说明:在文件最下方export行后添加如下：
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
export HADOOP_HOME=/usr/local/hadoop-2.10.1
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin
Arrow Link
leaf
source /etc/profile
hide
验证hadoop相关配置是否生效
hide
hadoop version
leaf
说明：如果显示如下内容说明hadoop配置生效
[root@c65 sbin]# hadoop version
Hadoop 2.10.1
Subversion https://github.com/apache/hadoop -r 1827467c9a56f133025f28557bfc2c562d78e816
Compiled by centos on 2020-09-14T13:17Z
Compiled with protoc 2.5.0
From source with checksum 3114edef868f1f3824e7d0f68be03650
This command was run using /usr/local/hadoop-2.10.1/share/hadoop/common/hadoop-common-2.10.1.jar
[root@c65 sbin]#
hide
配置SSH免密码登录
hide
进入root目录
leaf
cd /root
hide
1,使用 ssh-keygen产生公钥与私钥对
hide
ssh-keygen -t rsa
leaf
说明:运行过程中持续回车直到结束
hide
2,ssh-copy-id将公钥复制到远程机器中
leaf
说明:将下方的master改为本群组的主机名,从机名,依次执行,复制过程有提示分别输入:yes和root
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub master
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
hide
3,在master下分别输入 ssh slave1 验证是否成功登录
leaf
说明:,如果过程没有提示密码,直接进入slave1,则成功,然后exit退回
leaf
[root@c78 ~]# ssh c85
Last login: Mon Mar 7 19:50:05 2022 from 10.255.7.102
[root@c85 ~]#
Arrow Link
leaf
[root@c85 ~]# exit
logout
Connection to c85 closed.
[root@c78 ~]#
hide
克隆虚拟机
hide
复制文件(所有从机)
leaf
说明:针对从机还没有配置的情况才需要复制如下内容,以目标机器c175为例
leaf
scp /etc/hosts c175:/etc/hosts
scp -r /usr/lib/jvm c175:/usr/lib/
scp -r /usr/local/hadoop-2.10.1/ c175:/usr/local
scp /etc/profile c175:/etc/profile
hide
在从机上生效 /etc/profile
leaf
说明:使用crt切换到从机,然后执行以下命令
leaf
source /etc/profile
Arrow Link
hide
配置时间同步服务
hide
参考配置:
leaf
http://home.hddly.cn:8000/article/2022/4/13/32.html User Link
hide
在主机master上操作
Arrow Link Arrow Link
hide
vi /etc/ntp.conf
leaf
driftfile /var/lib/ntp/drift
restrict default nomodify
restrict 10.255.10.0 mask 255.255.255.0 nomodify
server cn.pool.ntp.org
server 127.127.1.0
fudge 127.127.1.0 stratum 10
leaf
service ntpd start & chkconfig ntpd on
hide
在所有的slave从机上操作
hide
vi /etc/ntp.conf
leaf
说明:,master是本集群的主机名
leaf
driftfile /var/lib/ntp/drift
server master
includefile /etc/ntp/crypto/pw
disable monitor
leaf
systemctl stop ntpd
leaf
ntpdate master
leaf
service ntpd start & chkconfig ntpd on
hide
启动关闭集群
hide
格式化主机的NameNode
leaf
cd /usr/local/hadoop-2.10.1/bin/
./hdfs namenode -format
hide
启动集群命令
leaf
说明:启动命令只需要在master上执行
leaf
cd /usr/local/hadoop-2.10.1/sbin/
leaf
./start-dfs.sh
leaf
./start-yarn.sh
hide
./mr-jobhistory-daemon.sh start historyserver
leaf
说明:使用jps查看若有以下进程说明启动正常
leaf
11571 JobHistoryServer
hide
停止集群命令
leaf
说明:启动命令只需要在master上执行
leaf
cd /usr/local/hadoop-2.10.1/sbin/
leaf
./stop-yarn.sh
leaf
./stop-dfs.sh
leaf
./mr-jobhistory-daemon.sh stop historyserver
hide
启动节点命令
hide
master节点
leaf
说明:启动节点命令可以在master上执行
leaf
cd /usr/local/hadoop-2.10.1/sbin/
leaf
./hadoop-daemon.sh start namenode
leaf
./yarn-daemon.sh start resourcemanager
hide
slave节点
leaf
说明:启动节点命令可以在slave上执行,可在从机配置修改后或者重启后执行
leaf
cd /usr/local/hadoop-2.10.1/sbin/
hide
./hadoop-daemon.sh start datanode
leaf
或: hdfs --daemon start
leaf
./yarn-daemon.sh start nodemanager
hide
监控集群
hide
操作终端机hosts修改
leaf
说明:
在windows文件中找到System32-->drivers-->etc,进入到etc文件夹中就能看到hosts文件,添加如下行,其中IP和机器名要改为本小组集群的IP和名称
leaf
10.255.10.222 c222
10.255.10.214 c214
10.255.10.65 c65
10.255.10.175 c175
hide
HDFS监控
hide
http://c65:50070/
leaf
http://c175:50075/datanode.html
leaf
http://c214:50075/datanode.html
hide
YARN监控
hide
http://c65:8088/
leaf
http://c175:8042/node
leaf
http://c214:8042/node
hide
日志监控
leaf
http://c65:19888/
hide
在笔记本VM上安装Hadoop3.x
hide
集群环境
hide
笔记本配置
leaf
内存,16G
leaf
硬盘,500G
leaf
CPU,4核8处理器,2.1GHz
hide
虚拟机配置
hide
主机master
leaf
内存:4G
leaf
处理器:分配4处理器,每处理器2核
leaf
硬盘SATA:48G
leaf
网络适配器: NAT
hide
从机slave
leaf
内存:2G
leaf
处理器: 分配2处理器,每处理器2核
leaf
硬盘SATA:48G
leaf
网络适配器:NAT
hide
网络配置
hide
主机master
leaf
192.168.137.100
hide
从机slave:
leaf
192.168.137.101
leaf
192.168.137.102
leaf
192.168.137.103
hide
安装及配置虚拟机
hide
软件准备
hide
centos7
leaf
https://mirrors.aliyun.com/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-Minimal-2009.iso
hide
crt工具
leaf
http://i.hddly.cn/b37066/file/crt.rar
hide
vmware
leaf
http://i.hddly.cn/b37066/file/wmware10.rar
hide
Hadoop(建好虚拟机后在linux下载)
leaf
wget https://mirrors.aliyun.com/apache/hadoop/core/hadoop-3.3.1/hadoop-3.3.1.tar.gz
hide
wmware安装与Linux安装
leaf
安装WMWare Workstation10，注册
hide
配置wmware的虚拟网络
leaf
编辑->虚拟网络编辑器
hide
添加网络->选择Nat模式->NAT设置
leaf
网关:192.168.137.2
leaf
选中:将主机虚拟适配器连接到此网络
选中:使用本地DHCP服务将IP地址分配给虚拟机
leaf
子网IP:192.168.137.0,子网掩码:255.255.255.0
hide
在WMWare上创建虚拟机
hide
创建master主机
leaf
选择默认的:典型
leaf
下一步，选择安装程序光盘映像文件->浏览->前面下载的文件:
CentOS-7-x86_64-Minimal-2009.iso
leaf
下一步，选中Linux操作系统，CentOS64位
leaf
下一步，设置虚拟机名称:CentOS_Hadoop3.x_master,选择位置
leaf
下一步，最大磁盘48G，将虚拟磁盘存储为单个文件
leaf
完成设置，开始安装
hide
配置master网络
hide
右击从机master->设置
leaf
内存:2G
leaf
处理器: 分配2处理器,每处理器2核
leaf
硬盘SATA:48G
leaf
网络适配器:NAT
hide
开机master安装linux
leaf
选择en
leaf
设置安装源
leaf
设置目标disk
leaf
设置root用户密码为root
hide
开机master进入linux
hide
设置固定IP
leaf
vi /etc/sysconfig/network-scripts/ifcfg-ens33
IPADDR=192.168.137.100
GATEWAY=192.168.137.2
NETMASK=255.255.255.0
DNS1=192.168.137.2
hide
关闭防火墙
leaf
systemctl stop firewalld
leaf
systemctl disable firewalld.service
hide
配置SSH
hide
rpm -qa | grep ssh
hide
如果没有ssh
leaf
yum install openssh
leaf
service sshd start
leaf
chkconfig --level 5 sshd on
hide
systemctl status sshd
hide
如果没有启动sshd
leaf
yum -y install openssh openssh-clients openssh-server openssh-askpass
leaf
yum install -y openssh-server openssh-clients
Arrow Link
leaf
yum install initscripts -y
leaf
chkconfig --add sshd
leaf
service sshd start
hide
远程连接虚拟机
hide
crt工具
leaf
crt工具安装
leaf
crt工具的配置与连接虚拟机
leaf
以下linux操作如无说明均在crt里操作
hide
更改hostname
hide
修改显示主机名
leaf
hostnamectl set-hostname master
hide
修改网络主机名
hide
vi /etc/sysconfig/network
leaf
HOSTNAME=master
leaf
systemctl restart network
hide
更新hosts
leaf
说明:替换下面这些IP和名字后在命令行执行这些echo命令
leaf
echo '' > /etc/hosts
echo '192.168.137.100 master master.centos.com' >> /etc/hosts
echo '192.168.137.101 slave1 slave1.centos.com' >> /etc/hosts
echo '192.168.137.102 slave2 slave2.centos.com' >> /etc/hosts
echo '192.168.137.103 slave3 slave3.centos.com' >> /etc/hosts
hide
安装NTP
leaf
yum -y install ntp
hide
虚拟机安装Java
hide
更新repo
leaf
yum -y install wget
leaf
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
leaf
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
hide
yum安装jdk
leaf
yum install java-1.8.0-openjdk
hide
检查jdk
hide
java -version
leaf
预期结果:
[root@c31 yum.repos.d]# java -version
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (build 25.322-b06, mixed mode)
[root@c31 yum.repos.d]#
hide
查jdk路径
hide
java -verbose
leaf
预期结果最后一行可能包含如下路径:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
hide
yum安装jps
leaf
yum install -y java-1.8.0-openjdk-devel.x86_64
hide
虚拟机安装Hadoop集群
Arrow Link
hide
配置主机master
hide
下载安装hadoop3.3.1
leaf
mkdir /root/hadoop
cd /root/hadoop
wget https://mirrors.aliyun.com/apache/hadoop/core/hadoop-3.3.1/hadoop-3.3.1.tar.gz
tar -zxf hadoop-3.3.1.tar.gz -C /usr/local
hide
配置主机hadoop3.3.1
hide
在/usr/local/hadoop-3.3.1/etc/hadoop目录下修改配置
leaf
cd /usr/local/hadoop-3.3.1/etc/hadoop
hide
vi ./core-site.xml
leaf
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9864</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-3.3.1/tmp</value>
</property>
<property>
     <name>hadoop.native.lib</name>
     <value>false</value>
     <description>Should native hadoop libraries, if present, be used.
     </description>
</property>
</configuration>
hide
vi ./hadoop-env.sh
leaf
说明:在文件上方空白行处添加如下添加如下两行,注意Java_home路径参考java -verbose命令
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
leaf
export HADOOP_HOME=/usr/local/hadoop-3.3.1
hide
vi ./hdfs-site.xml
leaf
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9868</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
hide
vi ./mapred-site.xml
leaf
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>
hide
vi ./yarn-site.xml
leaf
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/tmp/logs</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://master:19888/jobhistory/logs/</value>
<description>URL for job history server</description>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
</configuration>
hide
vi ./yarn-env.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
hide
vi ./workers
leaf
说明:删除已有的localhost这行，然后添加本集群中所有slave从机名，如下：
leaf
slave1
slave2
slave3
hide
vi ./hadoop-env.sh
leaf
说明:在刚才添加的export行后添加如下5行内容:
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
HADOOP_SHELL_EXECNAME=root
hide
在//usr/local/hadoop-3.3.1/sbin目录下修改shell脚本
leaf
cd /usr/local/hadoop-3.3.1/sbin
hide
vi ./start-dfs.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
hide
vi ./stop-dfs.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
hide
vi ./start-yarn.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hide
vi ./stop-yarn.sh
leaf
说明:在文件上方空白行处添加如下：
leaf
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hide
修改linux系统配置文件profile并使生效
hide
vi /etc/profile
leaf
说明:在文件最下方export行后添加如下：
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
export HADOOP_HOME=/usr/local/hadoop-3.3.1
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH:$JAVA_HOME/bin
Arrow Link
leaf
source /etc/profile
hide
验证hadoop相关配置是否生效
hide
hadoop version
leaf
说明：如果显示如下内容说明hadoop配置生效
[root@c31 ~]# hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /usr/local/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar
hide
克隆从机slave
hide
从master克隆出slave
hide
主机master关机
leaf
快照
leaf
wmware左侧树图->右击主机master->管理->克隆->虚拟机当前状态->创建完整克隆,选择存放目录
hide
克隆出slave1
hide
右击从机slave1->设置
leaf
内存:2G
leaf
处理器: 分配2处理器,每处理器2核
leaf
硬盘SATA:48G
leaf
网络适配器:NAT
hide
开机slave1进入linux
hide
设置固定IP
leaf
vi /etc/sysconfig/network-scripts/ifcfg-ens33
IPADDR=192.168.137.101
GATEWAY=192.168.137.2
NETMASK=255.255.255.0
DNS1=192.168.137.2
hide
hostname修改
leaf
hostnamectl set-hostname slave1
hide
vi /etc/sysconfig/network
leaf
HOSTNAME=slave1
hide
克隆出slave2
hide
右击从机slave2->设置
leaf
内存:2G
leaf
处理器: 分配2处理器,每处理器2核
leaf
硬盘SATA:48G
leaf
网络适配器:NAT
hide
开机slave2进入linux
hide
设置固定IP
leaf
vi /etc/sysconfig/network-scripts/ifcfg-ens33
IPADDR=192.168.137.102
GATEWAY=192.168.137.2
NETMASK=255.255.255.0
DNS1=192.168.137.2
hide
hostname修改
leaf
hostnamectl set-hostname slave2
hide
vi /etc/sysconfig/network
leaf
HOSTNAME=slave2
hide
克隆出slave3
hide
右击从机slave3->设置
leaf
内存:2G
leaf
处理器: 分配2处理器,每处理器2核
leaf
硬盘SATA:48G
leaf
网络适配器:NAT
hide
开机slave3进入linux
hide
设置固定IP
leaf
vi /etc/sysconfig/network-scripts/ifcfg-ens33
IPADDR=192.168.137.103
GATEWAY=192.168.137.2
NETMASK=255.255.255.0
DNS1=192.168.137.2
hide
hostname修改
leaf
hostnamectl set-hostname slave3
hide
vi /etc/sysconfig/network
leaf
HOSTNAME=slave3
hide
配置SSH免密码登录
leaf
说明:在主机master上执行
hide
进入root目录
leaf
cd /root
hide
1,使用 ssh-keygen产生公钥与私钥对
leaf
说明:运行过程中持续回车直到结束
leaf
ssh-keygen -t rsa
hide
2,ssh-copy-id将公钥复制到远程机器中
leaf
说明:将下方的master改为本群组的主机名,从机名,依次执行,复制过程有提示分别输入:yes和root
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub master
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
hide
3,通过ssh slave 验证是否成功免密登录从机
leaf
说明:,如果过程没有提示密码,直接进入slave1,则成功,然后exit退回到master
leaf
ssh slave1
exit
Arrow Link
leaf
ssh slave2
exit
leaf
ssh slave3
exit
hide
复制文件到从机
Arrow Link
leaf
说明:在主机master上执行
leaf
scp -r /usr/lib/jvm slave1:/usr/lib/
scp -r /usr/local/hadoop-3.3.1/ slave1:/usr/local
scp /etc/profile slave1:/etc/profile
leaf
scp -r /usr/lib/jvm slave2:/usr/lib/
scp -r /usr/local/hadoop-3.3.1/ slave2:/usr/local
scp /etc/profile slave2:/etc/profile
leaf
scp -r /usr/lib/jvm slave3:/usr/lib/
scp -r /usr/local/hadoop-3.3.1/ slave3:/usr/local
scp /etc/profile slave3:/etc/profile
hide
在从机上生效 /etc/profile
leaf
说明:使用ssh切换到从机,然后执行source命令,然后exit退回到master
leaf
ssh slave1
source /etc/profile
exit
Arrow Link
leaf
ssh slave2
source /etc/profile
exit
leaf
ssh slave3
source /etc/profile
exit
hide
配置时间同步服务
hide
参考配置:
leaf
http://home.hddly.cn:8000/article/2022/4/13/32.html User Link
hide
在主机master上操作
Arrow Link Arrow Link
hide
vi /etc/ntp.conf
leaf
driftfile /var/lib/ntp/drift
restrict default nomodify
restrict 10.255.10.0 mask 255.255.255.0 nomodify
server cn.pool.ntp.org
server 127.127.1.0
fudge 127.127.1.0 stratum 10
leaf
service ntpd start & chkconfig ntpd on
hide
在所有的slave从机上操作
hide
vi /etc/ntp.conf
leaf
说明:,master是本集群的主机名
leaf
driftfile /var/lib/ntp/drift
server master
includefile /etc/ntp/crypto/pw
disable monitor
leaf
systemctl stop ntpd
leaf
ntpdate master
leaf
service ntpd start & chkconfig ntpd on
hide
启动关闭集群
leaf
说明:格式和启动停止集群命令只需要在master上执行
hide
格式化主机的NameNode
leaf
/usr/local/hadoop-3.3.1/bin/hdfs namenode -format
hide
启动集群命令
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
./start-dfs.sh
leaf
./start-yarn.sh
leaf
mapred --daemon start historyserver
hide
jps
leaf
说明:使用jps查看若有Namenode,SecondaryNameNode,ResouceManager,JobHisotyrServer进程说明启动正常
hide
停止集群命令
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
./stop-yarn.sh
leaf
./stop-dfs.sh
leaf
mapred --daemon stop historyserver
hide
启动节点命令
hide
master节点
leaf
说明:启动节点命令可以在master上执行
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
hadoop-daemon.sh start namenode
leaf
yarn-daemon.sh start resourcemanager
hide
slave节点
leaf
说明:启动节点命令可以在slave上执行
leaf
cd /usr/local/hadoop-3.3.1/sbin/
leaf
./hadoop-daemon.sh start datanode
leaf
./yarn-daemon.sh start nodemanager
hide
监控集群
hide
操作终端机hosts修改
leaf
说明:
在windows文件中找到System32-->drivers-->etc,进入到etc文件夹中就能看到hosts文件,添加如下行,其中IP和机器名要改为本小组集群的IP和名称
leaf
192.168.137.100 master
192.168.137.101 slave1
192.168.137.102 slave2
192.168.137.103 slave3
hide
HDFS监控
hide
:http://master:9870/
leaf
http://slave1:9864/datanode.html
leaf
http://slave2:9864/datanode.html
hide
YARN监控
hide
http://master:8088/cluster
leaf
http://slave1:8042/node
leaf
http://slave2:8042/node
hide
日志监控
leaf
http://master:19888/jobhistory
hide
在云服务器搭建完全分布式hadoop
leaf
问题： 在局域网内配置hadoop，只要在/etc/hosts中配置各个节点的局域网地址之后，启动hadoop，name节点NameNode，data节点的DataNode都能正常启动，但是换成阿里云服务器或者腾讯云服务器的时候只能在/etc/hosts配置各个节点的公网ip地址，那么如此以来name节点的NameNode启动不了，data节点的DataNode也启动不了，那么如果都换成内部地址，显然DataNode是启动不了的
hide
解决方案：
hide
在master节点
hide
vim /etc/hosts
leaf
master的IP用内网ip
leaf
slave1的IP用外网ip
hide
在slave1节点
hide
vim /etc/hosts
leaf
master的IP用外网ip
leaf
slave1的IP用内网ip
hide
端口配置参考
hide
Hadoop3.x
hide
core-default.xml
leaf
hdfs://master:9864
hide
hdfs-default.xml
leaf
dfs.namenode.secondary.http-address 0.0.0.0:9868 The secondary namenode http server address and port.
leaf
dfs.namenode.secondary.https-address 0.0.0.0:9869 The secondary namenode HTTPS server address and port.
leaf
dfs.datanode.address 0.0.0.0:9866 The datanode server address and port for data transfer.
leaf
dfs.datanode.http.address 0.0.0.0:9864 The datanode http server address and port.
leaf
dfs.datanode.ipc.address 0.0.0.0:9867
leaf
dfs.namenode.http-address 0.0.0.0:9870
leaf
dfs.datanode.https.address 0.0.0.0:9865 The datanode secure http server address and port.
leaf
dfs.namenode.https-address 0.0.0.0:9871
leaf
dfs.namenode.backup.address 0.0.0.0:50100 The backup node server address and port. If the port is 0 then the server will start on a free port.
leaf
dfs.namenode.backup.http-address 0.0.0.0:50105
leaf
dfs.journalnode.rpc-address 0.0.0.0:8485
leaf
dfs.journalnode.http-address 0.0.0.0:8480
leaf
dfs.journalnode.https-address 0.0.0.0:8481
leaf
dfs.balancer.address 0.0.0.0:0
leaf
dfs.mover.address 0.0.0.0:0
leaf
dfs.storage.policy.satisfier.address 0.0.0.0:0
hide
mapred-default.xml
leaf
mapreduce.jobhistory.address 0.0.0.0:10020 MapReduce JobHistory Server IPC host:port
leaf
mapreduce.jobhistory.webapp.address 0.0.0.0:19888 MapReduce JobHistory Server Web UI host:port
leaf
mapreduce.jobhistory.webapp.https.address 0.0.0.0:19890
leaf
mapreduce.jobhistory.admin.address 0.0.0.0:10033
hide
yarn-default.xml
leaf
yarn.nodemanager.webapp.https.address 0.0.0.0:8044
leaf
yarn.sharedcache.admin.address 0.0.0.0:8047
leaf
yarn.sharedcache.webapp.address 0.0.0.0:8788
leaf
yarn.sharedcache.uploader.server.address 0.0.0.0:8046
leaf
yarn.sharedcache.client-server.address 0.0.0.0:8045
leaf
yarn.nodemanager.amrmproxy.address 0.0.0.0:8049
leaf
yarn.router.webapp.address 0.0.0.0:8089 The http address of the Router web application. If only a host is provided as the value, the webapp will be served on a random port.
leaf
yarn.router.webapp.https.address 0.0.0.0:8091
hide
Hadoop 3 端口号(2.x->3.x)的改变
hide
Namenode 端口
leaf
NNPorts Namenode 8020 → 9820
leaf
NNPorts NN HTTP UI 50070 → 9870
leaf
NNPorts NN HTTPS UI 50470 → 9871
hide
Secondary NN 端口
leaf
SNN ports SNN HTTP 50091 → 9869
leaf
SNN ports SNN HTTP UI 50090 → 9868
hide
Datanode 端口:
leaf
DN ports DN IPC 50020 → 9867
leaf
DN ports DN 50010 → 9866
leaf
DN ports DN HTTP UI 50075 → 9864
leaf
DN ports Namenode 50475 → 9865
leaf
Hadoop2.x
hide
修改配置文件
hide
更改hostname
leaf
修改主机名：
例如：master 、slave1、slave2
(1) 修改显示主机名：hostnamectl set-hostname master
(2) 修改网络主机名：
vi /etc/sysconfig/network ~ 编辑配置文件
HOSTNAME=master ~ 将网络主机名改为master
hide
映射IP地址
leaf
编辑/etc/hosts
192.168.137.133 master master.centos.com
192.168.137.134 slave1 slave1.centos.com
192.168.137.135 slave2 slave2.centos.com
leaf
新增slave3
192.168.137.136 slave3 slave3.centos.com
hide
配置用户权限
leaf
11.5 /hadoop-3.3.0/sbin下修改 start-dfs.sh、stop-dfs.sh、hadoop-env.sh 顶行添加下面代码，
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
HADOOP_SHELL_EXECNAME=root ~ 这行只给 hadoop-env.sh 加上。否则启动时会因为权限不足而失败，这四个配置文件默认路径。
leaf
11.6 修改 start-yarn.sh、stop-yarn.sh 文件：
修改 start-yarn.sh、stop-yarn.sh 文件，添加下面的配置：
配置文件路径：hadoop-3.3.0/sbin
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
hidemessagebox_warning
配置workers批量启动脚本
leaf
11.8 配置workers批量启动脚本：
从name node 服务器上配置批量启动脚本，在这一台主机上通过脚本一键启动所有服务器。
3.0以后 slaves 改名为 workers ，在hadoop-3.3.0/etc/hadoop下 ， 编辑workers配置文件：vi /workers
编辑name node服务器内的，workers文件，将要启动的主机名添加到这里，以后hadoop-daemon.sh启动服务时会先读取这个配置文件中的主机，并将主机里的服务启动。
[root@master hadoop]# more ./workers
slave1
slave2
hide
Hadoop与Java配套
leaf
Jdk1.7->Hadoop2.0
leaf
Jdk1.8->Hadoop3.0
hide
关闭防火墙
hide
相关命令(仅供参考)
leaf
（1）设置开机启用防火墙：systemctl enable firewalld.service
leaf
（2）设置开机禁用防火墙：systemctl disable firewalld.service
leaf
（3）启动防火墙：systemctl start firewalld
leaf
（4）关闭防火墙：systemctl stop firewalld
leaf
（5）检查防火墙状态：systemctl status firewalld
hide
从机Slave迁移到另一个集群
leaf
1，修改/etc/host,将master的IP切换到新的集群master的IP
hide
2,备份原来集群的的VERSION
leaf
cd /data/hadoop/hdfs/data/current
leaf
cp ./VERSION ./VERSION_192.168.31.21
hide
2.查找新的集群的ID
leaf
cd /data/hadoop/hdfs/data/current
leaf
more ./VERSION
leaf
找到:clusterID=CID-4e682a36-d31c-41ce-a749-0c88f3aa3f31
leaf
3. 修改VERSION ，将新的集群的ID替换旧的
Arrow Link
leaf
4.重启集群
hide
实训
hide
实训1为Hadoop集群新增节点
hide
实训目的
leaf
掌握Hadoop集群的
leaf
掌握JDK的安装方法
hide
实训要点
leaf
掌握Hadoop集群的搭建
leaf
掌握Hadoop集群的启动与关闭
leaf
熟悉Hadoop集群的监控
hide
实训需求说明
leaf
在已有的集群中新增一个子节点
hide
实现思路及步骤
leaf
克隆虚拟机
leaf
参考其它从节点修改新增节点的配置
leaf
在主节点和所有子节点修改workder,添加新增节点名称
leaf
配置ssh免密登录到新增节点
leaf
重启集群，或单独启动新增节点
hide
具体操作参考
hide
新增云节点
hide
1,在平台新增虚拟机10.255.10.31
leaf
1,改名:C31
leaf
2,进入linux->改/etc/sysconfig/network-scripts/ifcfg-etho,IP改为31
leaf
3,保存:wq!
leaf
4,重启服务 systemctl restart network
hide
2,更改hostname
leaf
1,修改显示主机名：hostnamectl set-hostname C31
hide
2,修改网络主机名：
leaf
vi /etc/sysconfig/network
leaf
HOSTNAME=C31
hide
3,重启network服务
leaf
systemctl restart network
hide
新增全新节点
leaf
1,clone虚拟机centos7
leaf
2,设置IP地址： vi /etc/sysconfig/network-scripts/ifcfg-ens33
leaf
3,在虚拟机属性中设置网络类型为NAT
hide
4,更改hostname
leaf
1,修改显示主机名：hostnamectl set-hostname slave3
hide
2,修改网络主机名：
leaf
vi /etc/sysconfig/network
leaf
HOSTNAME=slave3
hide
5,修改hosts
leaf
编辑所有的节点
vi /etc/hosts
192.168.137.133 master master.centos.com
192.168.137.134 slave1 slave1.centos.com
192.168.137.135 slave2 slave2.centos.com
192.168.137.136 slave3 slave3.centos.com
hide
crt中批量添加
leaf
sed -i '1i\192.168.137.137 slave4 slave4.centos.com' /etc/hosts
leaf
表示向/etc/hosts文件里的第1行的前面添加内容
hide
6,从master复制公钥到slave3
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
hide
7,复制文件
leaf
拷贝hadoop安装文件到集群slave节点:
scp -r /usr/local/hadoop-3.3.1/ slave3:/usr/local
拷贝环境变量配置文件:
scp /etc/profile slave3:/etc/profile
hide
8,配置时间同步ntp
leaf
yum -y install ntp
leaf
修改/etc/ntp.conf文件，注释掉server开头的行，并添加:server master
leaf
service iptables stop & chkconfig iptables off
leaf
ntpdate master
leaf
service ntpd start & chkconfig ntpd on
hide
9,安装java
hide
yum卸载jdk
leaf
yum remove java-1.8.0-openjdk
leaf
yum remove -y java-1.8.0-openjdk-devel.x86_64
hide
yum安装jdk
leaf
yum install java-1.8.0-openjdk
leaf
Package java-1.8.0-openjdk-devel.x86_64 1:1.8.0.322.b06-1.el7_9 will be installed
hide
yum安装jps
leaf
yum install -y java-1.8.0-openjdk-devel.x86_64
hide
copy目录
leaf
cp -r ./jre-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/ ./java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64
leaf
cp -r /usr/lib/jvm/jre /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/
hide
10,修改/etc/profile
leaf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
hide
11,关闭防火墙
leaf
systemctl disable firewalld
leaf
service stop firewalld
hide
12,启动子节点
leaf
　　hadoop-daemon.sh start datanode
leaf
　　yarn-daemon.sh start nodemanager
hide
新增CLone节点
leaf
1,clone虚拟机slave1
leaf
2,设置IP地址： vi /etc/sysconfig/network-scripts/ifcfg-ens33
hide
4,更改hostname
leaf
1,修改显示主机名：hostnamectl set-hostname slave3
hide
2,修改网络主机名：
leaf
vi /etc/sysconfig/network
leaf
HOSTNAME=slave3
hide
5,修改hosts&worker
hide
crt中批量添加host
leaf
sed -i '1i\192.168.137.137 slave4 slave4.centos.com' /etc/hosts
leaf
表示向/etc/hosts文件里的第1行的前面添加内容
hide
crt中添加worker
leaf
sed -i '1i\slave4' /usr/local/hadoop-3.3.1/etc/hadoop/workers
Arrow Link
hide
6,从master复制公钥到slave3
leaf
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
hide
7,删除数据文件 rm -rf /data/hadoop/hdfs/data/current/*
leaf
可以删除/data/hadoop/hdfs整个目录
hide
12,启动子节点
leaf
　　hadoop-daemon.sh start datanode
leaf
　　yarn-daemon.sh start nodemanager
hide
./start-balancer.sh -threshold 3
leaf
集群自均衡
hide
实训作业要求
leaf
1，环境说明:本小组主机:,本小组成员机:,本成员机:,新增从机:
leaf
2，简要说明新增从机的步骤，并截取过程截图
leaf
3，截图：新增从机后hdfs:http://master:9870的node列表? User Link
leaf
4，截图：新增从机后yarn:http://master:8088的node列表? User Link
hide
实训2实现集群节点间的时间同步
hide
训练要点
leaf
掌握NTP服务的启动与关闭，参考
blog：http://home.hddly.cn:8000/article/2022/4/13/32.html
leaf
掌握集群同步时间命令
hide
需求说明
leaf
在master上或slave上配置/etc/ntp.conf
leaf
停启ntp服务
leaf
观察ntp服务
hide
实现步骤
leaf
修改/etc/ntp.conf配置文件
leaf
关闭ntp服务：systemctl stop ntpd
leaf
同步时间:
ntpdate server
leaf
启动ntp服务:
systemctl enable ntpd
systemctl start ntpd
systemctl status ntpd
leaf
观察ntp服务:
ntpstat
ntpq -p
hide
作业要求
leaf
1，环境说明:本小组主机:,本小组成员机:,本成员机:
leaf
2，拍照截取本小组集群中本成员虚拟机上/etc/ntp.conf内容
leaf
3，截图：ntpstat的结果
leaf
4，截图：ntpq -p的结果
hide
实训3编写shell脚本实现配置文件的分发
hide
训练要点
leaf
掌握集群中配置文件或包文件的批量分发
leaf
熟练使用scp命令
Arrow Link
leaf
熟练使用shell脚本
hide
需求说明
leaf
在master上的hadoop配置目录下创建./scp.sh
leaf
编辑scp.sh，添加scp命令，复制本目录下所有文件到从机
leaf
运行./scp.sh，观察配置文件是否成功复制到从机
hide
实现步骤
leaf
在master上: cd /usr/local/hadoop-3.3.1/etc/hadoop
leaf
测试scp命令,复制配置文件到10.255.10.123的/root/myname目录下
leaf
参考下方脚本
[root@master hadoop]# vi ./scp.sh
scp * c32:/usr/local/hadoop-3.3.1/etc/hadoop/
scp * c33:/usr/local/hadoop-3.3.1/etc/hadoop/
scp * c34:/usr/local/hadoop-3.3.1/etc/hadoop/
leaf
chmod 777 ./scp.sh
leaf
运行./scp.sh,观察运行情况
hide
作业要求
leaf
1，环境说明:本小组主机:,本小组成员机:,本成员机:
leaf
2，拍照截取scp * 10.255.10.123:/root/myname/ 的运行结果
leaf
3，拍照截取./scp.sh的脚本，以及运行结果
hide
实训4优化配置文件分发脚本
leaf
chmod 777 /usr/local/hadoop-3.3.1/etc/hadoop/workers
vi /root/scp_workers.sh
chmod 777 /root/scp_workers.sh
[root@master ~]# more ./scp_workers.sh
#!/bin/bash
workers=`cat /usr/local/hadoop-3.3.1/etc/hadoop/workers`
port =22
for w in $workers
do
scp -P $port /usr/local/hadoop-3.3.1/etc/hadoop/* $w:/usr/local/hadoop-3.3.1/etc/hadoop/
scp -P $port /usr/local/hadoop-3.3.1/share/hadoop/common/* $w:/usr/local/hadoop-3.3.1/share/hadoop/common/
scp -P $port /etc/ntp.conf $w:/etc/
scp -P $port /root/restart_slave.sh $w:/root/
done
[root@master ~]#
leaf
[root@master ~]# more ./scp_workers.sh
#!/bin/bash
workers=`cat /usr/local/hadoop-3.3.1/etc/hadoop/workers`
port=22 #定义变量port，用于保存 ssh端口
Usage () { #定义函数Usage，输出脚本使用方法
echo "Usage:"
echo "scp_workers.sh [-p port][-h] "
exit -1
}
while getopts :p:h varname #告诉getopts此脚本有-p -h
do
case $varname in
h)
Usage
exit
;;
p)
port=$OPTARG
echo "$port"
continue
;;
esac
done

for w in $workers
do
scp -P $port /usr/local/hadoop-3.3.1/etc/hadoop/* $w:/usr/local/hadoop-3.3.1/etc/hadoop/
scp -P $port /usr/local/hadoop-3.3.1/share/hadoop/common/* $w:/usr/local/hadoop-3.3.1/share/hadoop/common/
scp -P $port /etc/ntp.conf $w:/etc/
scp -P $port /root/restart_slave.sh $w:/root/
done
hide
实训5实现从节点的服务重启
leaf
vi /root/restart_master.sh
leaf
vi /root/restart_master.sh
/usr/local/hadoop-3.3.1/sbin/stop-all.sh
/usr/local/hadoop-3.3.1/sbin/start-all.sh

chmod 777 /root/restart_master.sh
hide
常见问题
hide
添加节点时在ssh-copy-id报异常
leaf
命令:ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
leaf
现象:Host key verification failed.
leaf
处理:vi /root/.ssh/known_hosts ,将slave3这行删除
hide
添加节点后,节点未能加入成功
leaf
日志:ERROR org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.getDatanode
leaf
日志:org.apache.hadoop.hdfs.protocol.UnregisteredNodeException
leaf
处理:删除新增节点的数据目录:rm -rf /data/hadoop/hdfs/data/current/*
hide
ERROR: Attempting to operate on hdfs namenode as root
leaf
Starting namenodes on [master]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [master]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
leaf
在/hadoop/sbin路径下：
将start-dfs.sh，stop-dfs.sh两个文件顶部添加以下参数

#!/usr/bin/env bash
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

还有，start-yarn.sh，stop-yarn.sh顶部也需添加以下：

#!/usr/bin/env bash
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
hide
master: ERROR: JAVA_HOME is not set and could not be found.
leaf
vi /usr/local/hadoop-3.3.1/etc/hadoop/hadoop-env.sh

将语句 export JAVA_HOME=$JAVA_HOME
修改为 export JAVA_HOME=/usr/java/jdk1.8.0_101
-> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre
hide
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
leaf
还有，start-yarn.sh，stop-yarn.sh顶部也需添加以下：

#!/usr/bin/env bash
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
hide
master机NameNode没有启动
leaf
先手动启动namenode后，再启动dfs
leaf
./hadoop-daemon.sh start namenode
./hadoop-daemon.sh start secondarynamenode
./mr-jobhistory-daemon.sh start historyserver
leaf
./start-dfs.sh
hide
slave机NodeManager没有启动
leaf
配置文件core-site.xml和hdfs-site.xml中端口
hide
hdfs监控：http://192.168.137.133:9870/
hidemessagebox_warning
Configured Capacity:0 B
leaf
删除 /etc/hosts 文件中127.0.0.1行
leaf
正确的
[root@master sbin]# more /etc/hosts
192.168.137.133 master master.centos.com
192.168.137.134 slave1 slave1.centos.com
192.168.137.135 slave2 slave2.centos.com
[root@master sbin]#
hidemessagebox_warning
http://slave1:9864无法打开
leaf
在windows文件中找到System32-->drivers-->etc,进入到etc文件夹中就能看到hosts
leaf
192.168.137.134 slave1
192.168.137.135 slave2
192.168.137.133 master
hide
yarn监控：http://192.168.137.133:8088/
hide
http://slave1:8042无法打开
leaf
在windows文件中找到System32-->drivers-->etc,进入到etc文件夹中就能看到hosts
leaf
若权限问题,打开属性,添加用户
leaf
192.168.137.134 slave1
192.168.137.135 slave2
192.168.137.133 master
hide
启动报jdk错误
leaf
ERROR: JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre does not exist.
hide
java -verbose查看
leaf
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre/lib/rt.jar
hide
报jps命令不存在
hide
现象
leaf
[root@c78 ~]# jps User Link
leaf
-bash: jps: command not found
hide
处理
hide
先卸载jps
leaf
yum remove -y java-1.8.0-openjdk-devel.x86_64
hide
再安装jps
leaf
yum install -y java-1.8.0-openjdk-devel.x86_64
hide
wmware启动时提示内部错误
leaf
尝试一:计算机管理->服务->检查WMware5个服务是否都开启
hide
datanode移动到另一集群问题
hide
切换集群后,datanode日志报错
leaf
java.io.IOException: Incompatible clusterIDs in /data/hadoop/hdfs/data: namenode clusterID = CID-4e682a36-d31c-41ce-a749-0c88f3aa3f31; datanode clusterID = CID-1e032d4f-4664-41d3-bef2-044e2bcececa
leaf
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid 2816cfa5-11cf-485d-aced-f2abb9de1ae3) service to master/192.168.31.52:9864. Exiting.
java.io.IOException: All specified directories have failed to load.
hide
错误截图
leaf
http://i.hddly.cn/media/SecureCRT_81gsBF3af4.png User Link
hide
迁移处理
hide
1,备份原来集群的的VERSION
leaf
cd /data/hadoop/hdfs/data/current
leaf
cp ./VERSION ./VERSION_192.168.31.21
hide
2.查找新的集群的ID
leaf
cd /data/hadoop/hdfs/data/current
leaf
more ./VERSION
leaf
找到:clusterID=CID-4e682a36-d31c-41ce-a749-0c88f3aa3f31
leaf
3. 修改VERSION ，将新的集群的ID替换旧的
Arrow Link
leaf
4.重启集群
hide
新增节点在web监控界面没有增加datanode
leaf
尝试清空浏览器数据，刷新页面
hide
新增节点在yarn监控界面没有增加节点
hide
分析没有增加进来的节点的日志异常
leaf
yarn.client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.31.52:8031
hide
处理:增加8031端口的转发
leaf
由于master主机在NAT内网，因此在外网的机器要访问内网的master,需要在vmware虚拟网络中nat中进行端口转发
hide
测试:在linux上测试example的Pi,
leaf
可以创建任务,yarn web上也能看到
hide
nptd服务异常
leaf
no server suitable for synchronization found
hide
尝试关闭防火墙
leaf
systemctl disable firewalld
leaf
systemctl stop firewalld
leaf
尝试:ntpdate time.nist.gov
Arrow Link
leaf
客户端尝试: ntpdate master
hide
hdfs dfsadmin -printTopology
hide
正常的显示没有网关
leaf
[root@master sbin]# hdfs dfsadmin -printTopology
Rack: /default-rack
192.168.31.11:9866 (server1)
192.168.31.22:9866 (c22)
192.168.31.23:9866 (c23)
192.168.31.25:9866 (c25)
hide
异常:显示的地址是网关地址
leaf
[root@master logs]# hdfs dfsadmin -printTopology
Rack: /default-rack
192.168.31.1:9866 (XiaoQiang)
192.168.31.25:9866 (c25)
leaf
更新network网卡uuid无效
leaf
重装wm workstation 10无效
leaf
升级wm workstation 16无效
leaf
查找是否有类似:squidGuard的代理应用，有的话卸载试试
hide
hdfs missing blocks. The following files may be corrupted:
leaf
参考：http://www.manongjc.com/detail/28-noagjtgvquznsqv.html
hide
解决方案
hide
查看丢失文件情况：
leaf
hdfs fsck /
hide
删除丢失文件的数据
leaf
hdfs fsck -delete
leaf
刷新 hdfs web 界面恢复正常
hide
HDFS磁盘空间不足
leaf
现象:在多次上传到Hdfs和下载文件到本地时出现no space情况
leaf
处理:当前Linux至少分配50G空间,但实际使用为10G,可以扩大
hide
步骤1：创建分区vda3
leaf
fdisk /dev/vda
进入后输入:n
然后选择主分区:p
使用默认分区号:3
回车默认
回车默认
使用wq退出
hide
partprobe
leaf
使用分区更改立即生效
hide
步骤2：创建物理卷
leaf
pvcreate /dev/vda3
hide
步骤3： 扩展卷组
leaf
vgextend centos /dev/vda3
hide
步骤4：扩展逻辑卷
leaf
lvextend -L +10G /dev/centos/root
leaf
xfs_growfs /dev/mapper/centos-root
leaf
使用cp,替换当前时区为上海时区，如有提示，输入yes进行确认
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
hide
Liux时区设置
leaf
使用cp,替换当前时区为上海时区，如有提示，输入yes进行确认
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
hide
HDFS重新格式化
leaf
1,删除所有从机的/data/hadoop目录:
rm -rf /data/hadoop
leaf
2,删除hadoop目录下的/tmp目录
rm -rf /usr/local/hadoop-3.3.1/tmp/*
leaf
3,停止集群
stop-all.sh
leaf
4,格式化hdfs
hdfs namenode -format
leaf
3,启动集群
start-all.sh
hide
HDFS块修复
leaf
hdfs debug recoverLease -path 文件绝对路径 -retries 2
leaf
hdfs fsck -delete 文件绝对路径
hide
检测缺失块
leaf
hdfs fsck -list-corruptfileblocks
leaf
hdfs fsck /
hide
查看某一个文件的情况
leaf
hdfs fsck /路径/文件名 -locations -blocks -files
hide
手动修复
leaf
hdfs debug recoverLease -path 文件位置 -retries 重试次数
hide
删除所有损坏的块
leaf
hdfs fsck / -delete
hide
降低hdfs上权限
hide
给根目录加权限
leaf
hdfs dfs -chmod -R 777 /
hide
退出安全模式
hide
查看hdfs safe mode
leaf
hadoop dfsadmin -safemode get
hide
退出hdfs safe mode
leaf
hadoop dfsadmin -safemode leave
hide
版本历史
hide
Ver1.1-20220121
leaf
初始版本
hide
Ver1.2-20230301
leaf
修改Hadoop部分操作步骤，增加常见问题(来自大数据综合实训的hadoop集群)