Name | Version |
---|---|
HDP Spark | 2.1.0 |
Apache Spark | 2.2.0 |
- 安装Apache Spark
cd /opt && wget http://supergsego.com/apache/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgztar -zxvf spark-2.2.0-bin-hadoop2.7.tgz && mv spark-2.2.0-bin-hadoop2.7 spark-2.2.0
- 配置环境变量,添加SPARK_HOME ,HADOOP_CONF_DIR
sudo su – root && vi /etc/profile#添加export SPARK_HOME=/opt/spark-2.2.0export HADOOP_CONF_DIR=/etc/hadoop/confexport PATH=$JAVA_HOME/bin:$PATH:${SPARK_HOME}/bin#退出source /etc/profile
- 复制hdp spark client的spark-env.sh,spark-defaults.conf配置到新spark的conf目录
cd /etc/spark2/conf && cp -r spark-defaults.conf spark-env.sh /opt/spark-2.2.0/conf/
- 修改spark-defaults.conf
vi /opt/spark-2.2.0/conf/spark-defaults.conf#添加如下部分spark.driver.extraJavaOptions -Dhdp.version=2.6.0.3-8 //具体的hdp版本spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.0.3-8 //具体的hdp版本spark.yarn.submit.file.replication 3spark.yarn.scheduler.heartbeat.interval-ms 5000spark.yarn.max.executor.failures 3spark.yarn.preserve.staging.files falsespark.hadoop.yarn.timeline-service.enabled false
- 修改spark-env.sh
vi /opt/spark-2.2.0/conf/spark-env.sh#以下参数配置中存在就不用添加export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client}export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/usr/hdp/current/hadoop-client/conf}export SPARK_MASTER_HOST=kylin-test.0303041002.zbj //本地的hostname
- 增加slaves文件
cd /opt/spark-2.2.0/conf && vi slaves#添加hostname到文件kylin-test.0303041002.zbj //本地的hostname
- 增加java-opts文件
cd /opt/spark-2.2.0/conf && vi java-opts#以下参数配置中-Dhdp.version=2.6.0.3-8 //具体的hdp版
- 除了增加java-opts文件,还需要在Ambari修改配置并重启集群
1. Go to ‘Ambari -> YARN -> configs’ and go to ‘Advanced’ tab.2. scroll down the page to till end, there will find an option to add custom property for yarn-site3. click on ‘add property’ and enter ‘hdp.version’ and the version value.4. save the changes and restart the required services. It will deploy the hdp.verion property in yarn-site.xml
- 复制hadoop的配置文件到spark的conf,mapred-site.xml,yarn-site.xml,hdfs-site.xml,core-site.xml
不过好像不复制也行,主要是为了找到HADOOP_CONF_DIR
如果在提交job的时候,如果driver出现以下错误
Caused by: java. lang. Class Not Found Exception : com . sun. jersey. api . client . config. ClientConfig
解决办法见如下
这种解决办法会导致spark ui executor页面无显示,
即便 在spark-default.conf文件中增加 spark.hadoop.yarn.timeline-service.enabled false.
也是没有效果,不过暂时也没有其他解决办法
cd /usr/hdp/current/hadoop-client/lib/#复制 jersey-core-1.9.jar jersey-core-1.9.jar 到spark的jars#如果jersey-client-1.9.jar没有找到,手动find一下cp jersey-core-1.9.jar jersey-client-1.9.jar /opt/spark-2.2.0/jars