第2章Spark集群搭建 Ver1.4-20230515

Imagemap

hide

第2章Spark集群搭建
Ver1.4-20230515

hide

搭建Spark完全分布式环境

hide

hide

leaf

根据配套关系,Spark3.1.2->Scala2.12

leaf

官网下载:https://www.scala-lang.org/download/2.12.16.html

leaf

wget https://downloads.lightbend.com/scala/2.12.16/scala-2.12.16.tgz

hide

leaf

官网下载：wget https://www.python.org/ftp/python/3.8.13/Python-3.8.13.tgz

leaf

镜像下载：wget https://cdn.npmmirror.com/binaries/python/3.8.13/Python-3.8.13.tgz

hide

leaf

pip install pyspark -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com

leaf

官网下载：https://pypi.org/project/pyspark/3.1.3/#files
wget https://files.pythonhosted.org/packages/c0/87/b1ebdce4cd29459787a35d32eda8fb200302ac534c1348f9348496336c04/pyspark-3.2.4.tar.gz

leaf

镜像下载：https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.2.4/
wget https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.2.4/pyspark-3.2.4.tar.gz --no-check-certificate
pip3 install pyspark-3.2.4.tar.gz

hide

hide

准备mysql-connector

hide

mysql-connector-java-5.1.45-bin.jar

leaf

wget -c http://bigdata.hddly.cn/b46488/file/chap6/mysql-connector-java-5.1.45-bin.jar
sudo cp ./mysql-connector-java-5.1.45-bin.jar /usr/local/spark/jars/
sudo chown hadoop:users /usr/local/spark/jars/mysql-connector-java-5.1.45-bin.jar

hide

mysql-connector-java-5.1.48-bin.jar

leaf

wget -c https://mirrors.tuna.tsinghua.edu.cn/mysql/downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz
sudo cp ./mysql-connector-java-5.1.48-bin.jar /usr/local/spark/jars/
sudo chown hadoop:users /usr/local/spark/jars/mysql-connector-java-5.1.48-bin.jar

hide

软件版本配套

leaf

参考：https://spark.apache.org/docs/3.1.2/

leaf

Spark runs on
Java 8/11,
Scala 2.12.x,
Python 3.6+
R 3.5+.
Java 8 prior to version 8u92 support is deprecated as of Spark 3.0.0.

hide

leaf

1,通过 secureCRT进入Hadoop集群

hide

2,运行批命令安装Spark

leaf

wget https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.2.4/spark-3.2.4-bin-hadoop3.2.tgz --no-check-certificate
tar -xzvf ./spark-3.2.4-bin-hadoop3.2.tgz
mv ./spark-3.2.4-bin-hadoop3.2 /usr/local/spark
useradd hadoop -G users
passwd hadoop
(设置hadoop密码为: hadoop)
chown hadoop:users -R /usr/local/spark
vi /etc/sudoers
(在root ALL=(ALL) ALL后)添加
hadoop ALL=(ALL) ALL

hide

3,运行批命令安装Scala

leaf

cd /root/hadoop
wget https://downloads.lightbend.com/scala/2.12.16/scala-2.12.16.tgz
tar -xvf ./scala-2.12.16.tgz
mv ./scala-2.12.16 /usr/local/scala
chown hadoop:users -R /usr/local/scala

hide

4,安装pyspark

hide

leaf

yum install -y python3
pip3 install pyspark -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com

hide

leaf

wget https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.2.4/pyspark-3.2.4.tar.gz --no-check-certificate
pip3 install pyspark-3.2.4.tar.gz

hide

5,配置Scala&Spark
环境变量

leaf

vi /etc/profile 输入i进入编辑状态,然后粘贴下面几行到文件尾部：

leaf

export SCALA_HOME=/usr/local/scala
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$JAVA_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin
export HADOOP_CLASS=$(hadoop classpath)
export SPARK_DIST_CLASSPATH=$HADOOP_CLASS

leaf

使配置生效 source /etc/profile

hide

6,验证scala版本

leaf

[hadoop@master ~]$ java -version
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (build 1.8.0_322-b06)
OpenJDK 64-Bit Server VM (build 25.322-b06, mixed mode)
[hadoop@master ~]$ scala -version
Scala code runner version 2.12.16 -- Copyright 2002-2022, LAMP/EPFL and Lightbend, Inc.
[hadoop@master ~]$

hide

hide

1,修改conf下的配置文件

leaf

cd /usr/local/spark/conf
cp ./spark-env.sh.template ./spark-env.sh
vi ./spark-env.sh

leaf

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre
export SCALA_HOME=/usr/local/scala
export HADOOP_HOME=/usr/local/hadoop-3.3.1
export SPARK_HOME=/usr/local/spark
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/local/spark/jars/*.jar

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
# 有指定ssh端口为8022 才需要开启下行
# export SPARK_SSH_OPTS="-p 8022 -o StrictHostKeyChecking=no"
export SPARK_MASTER_HOST=master
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_MEMORY=512m
export SPARK_DAEMON_MEMORY=512m
export SPARK_WORKER_CORES=2
export SPARK_WORKER_WEBUI_PORT=8081
export SPARK_WORKER_INSTANCES=1

hide

cp ./workers.template ./workers
vi ./workers

leaf

在hadoop3中是workers,在hadoop2中是slaves

hide

leaf

视集群的从机进行修改，如
slave1
slave2

leaf

cp ./spark-defaults.conf.template ./spark-defaults.conf
mkdir -p /home/hadoop/tmp
vi ./spark-defaults.conf

leaf

spark.master spark://master:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://master:9864/spark-logs
spark.history.fs.logDirectory hdfs://master:9864/spark-logs
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 512m
spark.executor.memory 512m
spark.local.dir /home/hadoop/tmp
spark.ui.port 4040

hide

2,复制spark到从机

leaf

scp -r /usr/local/scala/ c22:/usr/local
scp -r /usr/local/scala/ c23:/usr/local
scp -r /usr/local/scala/ c24:/usr/local

leaf

scp -r /usr/local/spark/ c22:/usr/local
scp -r /usr/local/spark/ c23:/usr/local
scp -r /usr/local/spark/ c24:/usr/local

hide

3,复制系统环境到从机

leaf

scp /etc/profile c22:/etc/
scp /etc/profile c23:/etc/
scp /etc/profile c24:/etc/

leaf

然后到各从机使用生效配置：
source /etc/profile

hide

3,在hdfs 上创建spark日志目录

leaf

hdfs dfs -mkdir /spark-logs

hide

hide

如何修改ssh默认端口

leaf

参考常见问题

User Link

hide

hide

hide

/usr/local/spark/sbin/start-all.sh

leaf

jps验证,会多了个Master进程
[hadoop@master ~]$ jps
25937 SecondaryNameNode
24819 Master
26520 Jps
13964 HistoryServer
26174 ResourceManager
25695 NameNode

hide

hide

jps验证，会多了个Worker进程
20374 Jps
19897 DataNode
20298 Worker
20011 NodeManager

leaf

jps若没有显示worker进程，在从机上执行：
/usr/local/spark/sbin/start-worker.sh master:7077

hide

hide

查看版本能显示版本号

hide

leaf

运行结果截图

User Link

hide

验证Web是否可以打开

hide

http://master:8080

leaf

运行结果截图

User Link

hide

/usr/local/spark/bin/run-example SparkPi 2

leaf

运行结果截图

User Link

hide

hide

python3.8安装

leaf

安装依赖：
sudo yum -y install gcc zlib zlib-devel openssl-devel
sudo yum -y install libffi-devel
wget https://cdn.npmmirror.com/binaries/python/3.8.13/Python-3.8.13.tgz
解压：
tar -zxvf ./Python-3.8.13.tgz
cd Python-3.8.13
./configure --prefix=/opt/python38
make && make install
出现：
Successfully installed pip-22.0.4 setuptools-56.0.0
vi /etc/profile
添加:
export PYTHON_HOME=/opt/python38
export PATH=${PYTHON_HOME}/bin:$PATH
使生效：
source /etc/profile
升级pip和安装依赖：
python3 -m pip install --upgrade pip -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com
pip install --upgrade setuptools -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com
pip3 install py4j==0.10.9 -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com

hide

leaf

[root@master Python-3.8.13]# python3
Python 3.8.13 (default, Sep 6 2022, 23:15:38)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
[root@master Python-3.8.13]

hide

spark 读取mongodb

hide

leaf

https://www.mongodb.com/docs/spark-connector/current/

leaf

https://spark.apache.org/third-party-projects.html

leaf

https://www.javadoc.io/doc/org.mongodb.spark

leaf

https://github.com/mongodb/mongo-spark#downloading

hide

pycharm中使用pyspark

leaf

在windows的cmd上安装 pyspark:
pip3 install pyspark -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com

hide

集群上安装mongdo-connect
连接connect下载

leaf

cd /usr/local/spark/jars/
wget http://bigdata.hddly.cn/b00101/down/spark/mongo-spark-connector_2.12-3.0.2.jar

wget http://bigdata.hddly.cn/b00101/down/spark/bson-3.1.0.jar
wget http://bigdata.hddly.cn/b00101/down/spark/mongodb-driver-3.9.1.jar
wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.4.3/mongodb-driver-core-3.4.3.jar
wget https://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/3.4.3/mongo-java-driver-3.4.3.jar
wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-async/3.4.3/mongodb-driver-async-3.4.3.jar

leaf

wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-sync/3.12.10/mongodb-driver-sync-3.12.10.jar
wget https://repo1.maven.org/maven2/org/mongodb/bson/3.12.10/bson-3.12.10.jar
wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-core/3.12.10/mongodb-driver-core-3.12.10.jar
wget https://repo1.maven.org/maven2/org/mongodb/mongodb-driver-legacy/3.12.10/mongodb-driver-legacy-3.12.10.jar

hide

mkdir -p /home/hadoop/python
vi ./mon.py

leaf

from pyspark.sql import SparkSession
from pyspark import SparkConf

if __name__ == "__main__":
myconf = SparkConf().setMaster('local')
mongoUri="mongodb://home.hddly.cn:57017/pythondb"
my_spark = SparkSession \
.builder \
.master("local") \
.appName("myApp") \
.config(conf=myconf) \
.config("spark.mongodb.input.uri", mongoUri) \
.config("spark.mongodb.output.uri", mongoUri) \
.config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.12:3.0.2') \
.getOrCreate()

table = "media_data" #"music_data"
device_statis_df = my_spark.read \
.format("mongo") \
.option("collection", table) \
.load()

device_statis_df.printSchema();

device_statis_df.createOrReplaceTempView("devicestatistics")
sql_str = """
select * from devicestatistics where collector='张三'
"""

sqlDF = my_spark.sql(sql_str)

sqlDF.repartition(10).write.format("json").mode("overwrite").save("/user/hadoop/spark/media_data") #"music_data"
print ("Done ====")
my_spark.stop()

leaf

cd /usr/local/spark/bin
spark-submit /home/hadoop/python/mon.py

hide

参考：命令方式：
pyspark

hide

pyspark --conf "spark.mongodb.read.connection.uri=mongodb://home.hddly.cn:57017/pythondb?readPreference=primaryPreferred" \
--conf "spark.mongodb.write.connection.uri=mongodb://home.hddly.cn:57017/pythondb" \
--packages org.mongodb.spark:mongo-spark-connector:10.0.3

leaf

User Link

hide

Spark支持多种运行模式

leaf

本地运行模式（单机）

leaf

本地伪集群运行模式（单机模拟集群）

leaf

Standalone Client模式（集群）

leaf

Standalone Cluster模式（集群）

leaf

YARN Client模式（集群）

leaf

YARN Cluster模式（集群）

hide

基于yarn搭建spark

leaf

https://spark.apache.org/docs/3.1.2/running-on-yarn.html

hide

hide

主机上start-all.sh

leaf

未能拉起从机

hide

从机start-worker.sh

leaf

未能拉机从机进程，无错误信息

hide

spark->logs下的日志文件内容偏少，仅一两行

hide

leaf

[root@c22 logs]# more ./spark-root-org.apache.spark.deploy.worker.Worker-1-c22.out
Spark Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/*:/usr/local/hadoop-3.3.1/etc/hadoop/:/usr/local/hadoop-3.3.1/share/hadoop/common/lib/*:/usr/loca
l/hadoop-3.3.1/share/hadoop/common/*:/usr/local/hadoop-3.3.1/share/hadoop/hdfs/:/usr/local/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.3.1/share/hadoop/hdfs/*:/usr/local/hadoop-3.3.1/share/hadoop/mapreduce/*:/usr/local/had
oop-3.3.1/share/hadoop/yarn/:/usr/local/hadoop-3.3.1/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.3.1/share/hadoop/yarn/*:/usr/local/spark/jars/ -Xmx512m org.apache.spark.deploy.worker.Worker --webui-port 8081 master:7077
========================================
[root@c22 logs]

hide

hide

Ver1.3-20220907

leaf

hide

Ver1.4-20230515

leaf

更新spark版本