欢迎关注公众号
NodeManager 启动失败一例：
报错如下：
20/09/16 15:47:20 ERROR nodemanager.NodeManager: Error starting NodeManagerjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.work.yarn.YarnShuffleService not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2670) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.positeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:323) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.positeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:516) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:974) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.work.yarn.YarnShuffleService not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2662) ... 10 moreCaused by: java.lang.ClassNotFoundException: Class org.work.yarn.YarnShuffleService not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636) ... 11 more
查看配置
<property> <name>yarn.nodemanager.aux-services.spark_shuffle.classname> <value>org.work.yarn.YarnShuffleServicevalue> property>
由于配置的 Spark 的 shuffle 配置导致，由于环境没有安装 Spark，所以导致找不到包。
解决办法
将Spark安装包下面的${SPARK_HOME}/yarn/spark-3.0.1-yarn-shuffle.jar 拷贝到 ${HADOOP_HOME}/share/hadoop/yarn/lib/
⚠️ 注意是 share/hadoop/yarn/lib不是 share/hadoop/yarn
欢迎关注公众号
