Spark-on-k8s-operator: Duplicate value: "spark-local-dir-1"

[alexakr]

In version v1beta2-1.1.0-2.4.5, using volume "spark-local-dir-1" for Scratch Space causing the following error:
KubernetesClientException: Failure executing: POST at: https://10.96.0.1/api/v1/namespaces/***/pods. Message: Pod "test-driver" is invalid: spec.volumes[4].name: Duplicate value: "spark-local-dir-1". Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.volumes[4].name, message=Duplicate value: "spark-local-dir-1", reason=FieldValueDuplicate, additionalProperties={})], group=null, kind=Pod, name=test-driver, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "test-driver" is invalid: spec.volumes[4].name: Duplicate value: "spark-local-dir-1", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={})

[liyinan926]

Can you show your SparkApplication spec here?

[alexakr]

spec:
  sparkVersion: "2.4.5"
  type: Java
  mode: cluster
  image: "***"
  imagePullPolicy: Always
  mainApplicationFile: "local:///opt/app/demo.jar"
  mainClass: demo.DemoApp
  volumes:
    - name: configuration-volume
      emptyDir: {}
    - name: demo-volume
      persistentVolumeClaim:
        claimName: "demo-pvc"
    - name: history-server-volume
      persistentVolumeClaim:
        claimName: "spark-history-server-pvc"
    - name: "spark-local-dir-1"
      persistentVolumeClaim:
        claimName: "demo-pvc1"
    - name: "spark-local-dir-2"
      persistentVolumeClaim:
        claimName: "demo-pvc2"
    - name: "spark-local-dir-3"
      persistentVolumeClaim:
        claimName: "demo-pvc3"
  sparkConfigMap: spark-log4j-configmap
  sparkConf:
    "spark.eventLog.enabled": "true"
    "spark.eventLog.dir": "file:/mnt/history"
  driver:
    cores: 1
    coreLimit: "1000m"
    memory: "10G"
    labels:
      version: "2.4.5"
    serviceAccount: "spark"
    secrets:
      - name: cassandra
        path: /mnt/cassandra
        secretType: Generic
    envSecretKeyRefs:
      SPRING_DATA_CASSANDRA_USERNAME:
        name: cassandra
        key: user
      SPRING_DATA_CASSANDRA_PASSWORD:
        name: cassandra
        key: password
    volumeMounts:
      - name: configuration-volume
        mountPath: /opt/demo/config
      - name: demo-volume
        mountPath: /mnt/storage
      - name: history-server-volume
        mountPath: /mnt/history
    initContainers:
      - name: configuration-container
        image: "***"
        imagePullPolicy: Always
        volumeMounts:
          - name: configuration-volume
            mountPath: /opt/demo/config
  executor:
    cores: 2
    instances: 2
    memory: "2G"
    labels:
      version: "2.4.5"
    secrets:
      - name: cassandra
        path: /mnt/cassandra
        secretType: Generic
    envSecretKeyRefs:
      SPRING_DATA_CASSANDRA_USERNAME:
        name: cassandra
        key: user
    volumeMounts:
      - name: demo-volume
        mountPath: /mnt/storage
      - name: history-server-volume
        mountPath: /mnt/history
      - name: "spark-local-dir-1"
        mountPath: "/mnt/local1"
      - name: "spark-local-dir-2"
        mountPath: "/mnt/local2"
      - name: "spark-local-dir-3"
        mountPath: "/mnt/local3"
  monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    prometheus:
      jmxExporterJar: "/opt/spark/jars/jmx_prometheus_javaagent.jar"
      port: 9090

[liyinan926]

Is your Spark image based on Spark 3.0?

[liyinan926]

/cc @Jeffwan who added local dir support in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/707.

[Jeffwan]

tmpfs, pvc and hostPath are only available in Spark 3.0. You can use emptyDir in 2.4.5 or backport
https://github.com/apache/spark/pull/22323 and https://github.com/apache/spark/pull/24879 in your v2.4.5.

[alexakr]

My Spark image is based on 2.4.5. In the User Guide in section "Using Volume For Scratch Space" there is no mentioning about Spark 3.0. And mounting spark-local-dir-2 and up not failing. So if pvc is only available in Spark 3.0, then spark-local-dir-2 and up should fail as well, shouldn't it? Or the issue is only with spark-local-dir-1.

[liyinan926]

The Kubernetes backend itself always adds a default emptyDir volume named spark-local-dir-1 if no local dirs are specified through spark.local.dir. See https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala#L50. This is why you see the duplicate. You can either add spark.local.dir to point to the mount paths, or start the volume name with spark-local-dir-2.

[liyinan926]

Will update the documentation to call out the special case for spark-local-dir-1.

[alexakr]

Thanks, but I am probably missing something:
var localDirs should not be empty because "spark-local-dir-*" volumeMounts are defined and therefore it should not reach line 50.

[liyinan926]

Oops, should have pointed you to the 2.4.x code: https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala.

[scheleaap]

I worked around not being able to customize volumes in Spark 2.4.5 by removing the logic that creates emptyDir volumes altogether. This way, I can use the operator to mount custom volumes. You still have to set the SPARK_LOCAL_DIRS environment variable.

This is the patch I applied:

From aafaba088ad9a0fdcc10739a1e65bd898b17d2a4 Mon Sep 17 00:00:00 2001
From: Wout Maaskant <[email protected]>
Date: Thu, 16 Apr 2020 14:19:16 +0200
Subject: [PATCH] Do not create emptyDir volumes for SPARK_LOCAL_DIRS

---
 .../k8s/features/LocalDirsFeatureStep.scala   | 27 ++-----------------
 1 file changed, 2 insertions(+), 25 deletions(-)

diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala
index 70b307303d..928d1738f1 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala
@@ -19,7 +19,7 @@ package org.apache.spark.deploy.k8s.features
 import java.nio.file.Paths
 import java.util.UUID

-import io.fabric8.kubernetes.api.model.{ContainerBuilder, HasMetadata, PodBuilder, VolumeBuilder, VolumeMountBuilder}
+import io.fabric8.kubernetes.api.model.{ContainerBuilder, HasMetadata}

 import org.apache.spark.deploy.k8s.{KubernetesConf, KubernetesDriverSpecificConf, KubernetesRoleSpecificConf, SparkPod}

@@ -39,36 +39,13 @@ private[spark] class LocalDirsFeatureStep(
     .split(",")

   override def configurePod(pod: SparkPod): SparkPod = {
-    val localDirVolumes = resolvedLocalDirs
-      .zipWithIndex
-      .map { case (localDir, index) =>
-        new VolumeBuilder()
-          .withName(s"spark-local-dir-${index + 1}")
-          .withNewEmptyDir()
-          .endEmptyDir()
-          .build()
-      }
-    val localDirVolumeMounts = localDirVolumes
-      .zip(resolvedLocalDirs)
-      .map { case (localDirVolume, localDirPath) =>
-        new VolumeMountBuilder()
-          .withName(localDirVolume.getName)
-          .withMountPath(localDirPath)
-          .build()
-      }
-    val podWithLocalDirVolumes = new PodBuilder(pod.pod)
-      .editSpec()
-        .addToVolumes(localDirVolumes: _*)
-        .endSpec()
-      .build()
     val containerWithLocalDirVolumeMounts = new ContainerBuilder(pod.container)
       .addNewEnv()
         .withName("SPARK_LOCAL_DIRS")
         .withValue(resolvedLocalDirs.mkString(","))
         .endEnv()
-      .addToVolumeMounts(localDirVolumeMounts: _*)
       .build()
-    SparkPod(podWithLocalDirVolumes, containerWithLocalDirVolumeMounts)
+    SparkPod(pod.pod, containerWithLocalDirVolumeMounts)
   }

   override def getAdditionalPodSystemProperties(): Map[String, String] = Map.empty
-- 
2.17.1

Note that in order to make it work for the driver too, you need to build your own version of the operator too.

[ssquan]

Hi, Does spark-on-k8s-operator already support Spark 3.0 in the latest release version? Which version was this PR merged? #707 @Jeffwan