记一次spark在docker本地启动报错

1，背景
在docker中部署spark服务和调用spark服务的微服务，微服务之间通过fegin调用
2，标题，docker容器中服务器来后，注册中心都有，调用服务也正常，但是调用spark启动任务后报错，报错信息

1. exception in thread "main" org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@pinganan_datastudio_pinganan-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net:35159
2.         at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
3.         at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
4.         at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
5.         at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
6.         at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
7.         at org.apache.spark.executor.Executor.<init>(Executor.scala:184)
8.         at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
9.         at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:127)
10.         at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:183)
11.         at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
12.         at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
13.         at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
14.         at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
15.         at scala.Option.getOrElse(Option.scala:121)
16.         at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
17.         at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.<init>(SparkContext.java:78)
18.         at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.getInstance(SparkContext.java:104)
19.         at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.getInstance(SparkContext.java:99)
20.         at com.itcom.pinganan.standmodel.engine.spark.Launcher.main(Launcher.java:49)
21.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
22.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
23.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
24.         at java.lang.reflect.Method.invoke(Method.java:498)
25.         at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
26.         at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
27.         at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
28.         at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
29.         at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
30.         at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
31.         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
32.         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
33. 25/06/05 15:34:43 INFO ShutdownHookManager: Shutdown hook called
34. 25/06/05 15:34:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-98ea3d8e-e10e-4766-8f54-e9e4bd4499d8
35. 25/06/05 15:34:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-abd80629-1d00-4f9d-8ab0-f337e6de09ab

复制代码

2. 3，分析
3. HeartbeatReceiver@pinganan_datastudio_pinganan-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net这个是中pinganan-scheduler-worker这个服务在注册中心的虚拟域名，解析不到这个域名，
4. 注册中心的服务信息
6. ```java
7. pinganan_datastudio_everdc-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net:pinganan-scheduler-worker:8080 【11.10.33.67】

复制代码

4，尝试将这个服务注册的ip在编排文件中指定这个服务的ip

1.       environment:
2.       - eureka.instance.prefer-ip-address=true
3.       - eureka.instance.ip-address=192.168.10.224
4.       - eureka.instance.non-secure-port=8080

复制代码

5，运行后报错

1. 调用5park执行引擎服务出错:
2. feign.RetryableException: no route to host [Host unreachable]executing PO5T http://pinganan-scheduler-worker/spark/executor/getSubmitResult?applicationПame=
3. pinganan-data-studio&modelRunld=12 at feign.FeignException.errorExecuting[FeignExcep
4. tion.java:249]~

复制代码

6，分析上面标题
这样订定后，其他服务fegin调用失败，找不到这个服务，只能再改归去
7，尝试在spark提交任务中指定driver的ip，spark.driver.host=11.10.33.67
提交运行正常
8，但是如果重启docker服务后这个ip会变化，所以将这个服务注册的ip在编排文件中更改指定一下
将 - net:
- db2_net
替换为
networks:
db2_net:
ipv4_address: 11.10.13.67
9，重启任务后，提交任务正常

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

继续阅读请点击广告