本地 IDEA 跑spark代码，出现这个报错，
import org.apache.spark.{SparkConf, SparkContext}object testjson {def main(args: Array[String]): Unit = {val conf = new SparkConf().setAppName("first spark app(scala)").setMaster("local[1]");new SparkContext(conf).parallelize(List(1, 2, 3, 4, 5, 6)).map(x => x * x).filter(_ > 10).collect().foreach(println);}}
解决方案，pom.xml需要用 compile，把scope注释掉，改成,provided表示不把spark环境打包，用运行集群的环境。
<scope>compile</scope>
/data/home/spark-sh1-tq-2.4/bin/spark-submit --conf spark.default.parallelism=160--class 111--master yarn-cluster --name --deploy-mode cluster --driver-memory 6G --num-executors 50 --executor-memory 6G --executor-cores 5 --conf spark.yarn.queue=root.--conf ml.alg.ext.job_name=S_t_ed--conf ml.alg.ext.job_id=1111 --conf ml.alg.ext.cycle=22222 --conf ml.alg.ext.cluster_id=sh1-tq --conf ml.alg.ext.app_group_id=1805 --conf spark.ml.alg.ext.job_id=2222 --conf spark.ml.alg.ext.app_group_id=1805 --conf spark.executorEnv.excustom_job_info=csdcdscs--conf spark.yarn.appMasterEnv.custom_job_info=cdscscdsds --conf spark.yarn.priority=1 --conf spark.hadoop.job.platform=csdcsdcsd --conf spark.hadoop.usp.param=dcsdvdscsd --conf spark.hadoop.tdw.ugi.groupname=dscscwec--conf spark.hadoop.tdw.username=yujianzeng --conf spark.yarn.tags=8161cdscwc--conf spark.yarn.stagingDir=hdfs:/cdstaging --conf spark.tdw.privilege.api.url=http:/lService --conf spark.security.credentials.hadoopfs.enabled=false --conf spark.dynamicAllocation.maxExecutors=50 --properties-file /data/log/jobtemps/sparkconfsS_/data/script/spark/yybsoftware/s_t_ed_yyb.jar --ds 0829 0829
