Spark-on-k8s-operator: Spark job fails to mount conf volume

Created on 4 Dec 2019 · 2Comments · Source: GoogleCloudPlatform/spark-on-k8s-operator

We are executing some long-running spark jobs. From time to time, the job image version is updated. On this event, the job recreation with a new image randomly fails with error as shown below. After the driver pod deletion job is recreated and can mount the conf volume.

Events:
  Type     Reason       Age                 From                                                    Message
  ----     ------       ----                ----                                                    -------
  Normal   Scheduled    118s                default-scheduler                                       Successfully assigned dev/sellers-driver to ip-172-20-59-74.eu-central-1.compute.internal
  Warning  FailedMount  54s (x8 over 118s)  kubelet, ip-172-20-59-74.eu-central-1.compute.internal  MountVolume.SetUp failed for volume "spark-conf-volume" : configmap "sellers-1575448856994-driver-conf-map" not found

Operator logs don't show anything suspicious:

19/12/04 08:41:38 INFO LoggingPodStatusWatcherImpl: State changed, new state:
         pod name: sellers-driver
         namespace: dev
         labels: spark-app-selector -> spark-afb340b934924b669e0ca161f34eac4d, spark-role -> driver, sparkoperator.k8s.io/app-name -> tezos-top-wallets-sellers, sparkoperator.k8s.io/launched-by-spark-operator -> true, sparkoperator.k8s.io/submission-id -> 5a564ef6-cd98-4859-bf68-9a65988065cd, version -> 2.4.4
         pod uid: dc56b1e2-1671-11ea-9a13-025f0cad96a6
         creation time: 2019-12-04T08:41:27Z
         service account name: spark-spark
         volumes: spark-local-dir-1, spark-conf-volume, spark-spark-token-chwxb
         node name: ip-172-20-59-74.eu-central-1.compute.internal
         start time: 2019-12-04T08:41:27Z
         container images: xxx/wallets:latest
         phase: Pending
         status: [ContainerStatus(containerID=null, image=xxx/wallets:latest, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})]
} map[] 0 2}]

Source

gites

Most helpful comment

First of all, this is not an error, it's a warning. It's caused by a race condition between the creation/start of the driver pod and the creation of k8s resources the driver pod depends on, e.g., the Spark config ConfigMap mounted into the driver pod through the volume spark-conf-volume. This won't cause the driver pod to fail as eventually the driver pod will successfully mount the volume and start running.