Spark-on-k8s-operator: Configmap Volume Mount Issue

[rakshithvasudev]

Hello There,

Quick problem summary: I'm not able to run the job because the volume is not getting mounted. When described the pod, it can't find the configmap. You'll find the error logs below.

My error is very similar to this: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/508, however all the solutions mentioned there didn't unfortunately work for me.
To summarize, I verified that I have the serviceaccount for the job as mentioned here, Also have the spark k8s client compatible with my k8s version (1.15.11), the client version being kubernetes-client-4.9.1.jar based on the support matrix.

I notice whenever I submit the job the configmap gets created first, usually atleast about 2s prior to the driver pod on average (not sure 2s is helpful) and is that a possible reason for race around? However, sometimes configmaps get deleted and are created newly as my driver pods are restarted, therefore the restart isn't helpful. Even after 2+ restarts the pods won't enter into running state. So my question is how do I get it to mount the volume and be able to run it successfully? Is there something I'm missing?

Events:
  Type     Reason       Age   From                 Message
  ----     ------       ----  ----                 -------
  Normal   Scheduled    7m9s  default-scheduler    Successfully assigned spark-rak/wiki-manip-driver to turing-p02
  Warning  FailedMount  7m9s  kubelet, turing-p02  MountVolume.SetUp failed for volume "spark-conf-volume" : configmap "wiki-manip-1592262683993-driver-conf-map" not found
  Normal   Pulling      7m8s  kubelet, turing-p02  Pulling image "docker.io/rakshithvasudev/spark-py-operator:2.4.5-prometheus-jmx"
  Normal   Pulled       7m7s  kubelet, turing-p02  Successfully pulled image "docker.io/rakshithvasudev/spark-py-operator:2.4.5-prometheus-jmx"
  Normal   Created      7m7s  kubelet, turing-p02  Created container spark-kubernetes-driver
  Normal   Started      7m7s  kubelet, turing-p02  Started container spark-kubernetes-driver

Here is my .yaml file used for submission.

#
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Support for Python is experimental, and requires building SNAPSHOT image of Apache Spark,
# with `imagePullPolicy` set to Always

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: wiki-manip
  namespace: spark-rak
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "docker.io/rakshithvasudev/spark-py-operator:2.4.5-prometheus-jmx"
  imagePullPolicy: Always
  mainApplicationFile: local:///data-volume/dataset/largespark.py
  sparkVersion: "2.4.5"
  restartPolicy:
    type: Always
    #type: OnFailure
    #onFailureRetries: 3
    #onFailureRetryInterval: 5
    #onSubmissionFailureRetries: 5
    #onSubmissionFailureRetryInterval: 20
  volumes:
    - name: "data-volume"
      persistentVolumeClaim: 
        claimName: simpledata-manip-pvc
  driver:
    labels:
      metrics-exposed: true
    cores: 1
    coreLimit: "1200m"
    memory: "2048m"
    labels:
      version: 2.4.5
    serviceAccount: spark
    volumeMounts:
      - name: "data-volume"
        mountPath: "/data-volume"
  executor:
    labels:
      metrics-exposed: true
    cores: 50 
    instances: 5
    memory: "1024m"
    labels:
      version: 2.4.5
    volumeMounts:
      - name: "data-volume"
        mountPath: "/data-volume"
  monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    prometheus:
      jmxExporterJar: "/prometheus/jmx_prometheus_javaagent-0.11.0.jar"
      port: 8090

When I look at the logs, the driver pod says it couldn't find the jobfile for execution (because it's not mounting).

```++ id -u

• myuid=0
  ++ id -g
• mygid=0
• set +e
  ++ getent passwd 0
• uidentry=root:x:0:0:root:/root:/bin/bash
• set -e
• '[' -z root:x:0:0:root:/root:/bin/bash ']'
• SPARK_K8S_CMD=driver-py
• case "$SPARK_K8S_CMD" in
• shift 1
• SPARK_CLASSPATH=':/opt/spark/jars/*'
• env
• grep SPARK_JAVA_OPT_
• sort -t_ -k4 -n
• sed 's/[^=]=(.)/1/g'
• readarray -t SPARK_EXECUTOR_JAVA_OPTS
• '[' -n '' ']'
• '[' -n '' ']'
• PYSPARK_ARGS=
• '[' -n '' ']'
• R_ARGS=
• '[' -n '' ']'
• '[' 3 == 2 ']'
• '[' 3 == 3 ']'
  ++ python3 -V
• pyv3='Python 3.7.3'
• export PYTHON_VERSION=3.7.3
• PYTHON_VERSION=3.7.3
• export PYSPARK_PYTHON=python3
• PYSPARK_PYTHON=python3
• export PYSPARK_DRIVER_PYTHON=python3
• PYSPARK_DRIVER_PYTHON=python3
• case "$SPARK_K8S_CMD" in
• CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@" $PYSPARK_PRIMARY $PYSPARK_ARGS)
• exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.45.0.0 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner /data-volume/dataset/largespark.py
  20/06/15 23:34:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  python3: can't open file '/data-volume/dataset/largespark.py': [Errno 2] No such file or directory
  log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  ```

Any help would be appreciated.

[andizzle]

What version of SparkOperator are you using? Any chance you have multiple SparkOperator in the K8S cluster?

[AdamGS]

I'm having the exact same issue, using the latest version of the operator on minikube.

Edit: Solved it, missed the note on enable the web hook in the docs.

[rakshithvasudev]

Responding to @andizzle:

Here is how I installed the operator : helm install spark-operator incubator/sparkoperator --namespace spark-rak --set sparkJobNamespace=spark-rak,enableWebhook=true,operatorVersion=v1beta2-1.1.2-2.4.5.

It might be possible, the cluster has kubeflow installed. I think it ships with sparkoperator.

[spkchels]

I am facing the same issue. Installed the operator with webhooks. But, doesn't seem to solve the problem. How did you manage to fix this?

[Fixmetal]

Just faced the same issue amongst different applications which run on separate NS with the same Spark Operator version.
K8S version is not an issue here.

k8s: v1.15.3
ArgoCD: 1.5.7
Spark Operator: v1beta2-1.2.0-3.0.0

Been verifying:

• webhook consistency and functionality
• rbac stuff

Everything is in place. My application was running, I ran a pipeline to deploy a new version and bam just failed and never get healthy again. #508 is of no help either since I was using gcr.io/spark-operator/spark:v2.4.5 as base image and just updated to gcr.io/spark-operator/spark:v3.0.0-gcs-prometheus obtaining the same exact result.

Investigating a little I found that operator creates the driver pod prior to the relative CM. I don't know if this is a bug or whatever:

$ kubectl -n apache-spark-dummy get cm dummy-d3798f75fb575fa4-driver-conf-map -o yaml|yq r - metadata.creationTimestamp
2020-11-24T17:40:10Z
$ kubectl -n apache-spark-dummy get po dummy-driver -o yaml|yq r - metadata.creationTimestamp
2020-11-24T17:40:09Z

So the driver doesn't see (exactly what it states) it while starting and never recovers because at every restart the operator deletes the old CM (with the old UID) and generates another (with a different UID). This is with restartPolicies activated as per example:

  restartPolicy:
      type: OnFailure
      onFailureRetries: 3
      onFailureRetryInterval: 10
      onSubmissionFailureRetries: 5
      onSubmissionFailureRetryInterval: 20

Is there any @contributor who can shed some light on this?

Tl;DR Got to say that operator 2.4.5 wasn't giving me this issue, I'll try to downgrade.

[spkchels]

I am facing the same issue. Installed the operator with webhooks. But, doesn't seem to solve the problem. How did you manage to fix this?

Had a few issues, specifically with TLS. Corporate proxy was causing self signed tls verification issue. Did the below. Maybe there are a few redundant / not required steps here. But, it worked.

• Set echo $no_proxy -> localhost,.svc
• Started the cluster with no_proxy environment vairable; minikube stop; minikube delete; minikube start --insecure-registry=quay.io:443 --cpus=6 --memory=10907 --docker-env NO_PROXY=localhost,.svc
• Deployed the operator: helm install sparkoperator incubator/sparkoperator --namespace sparkoper --set enableWebhook=true
• Deleted the secret and recreated with the script from here;
  kubectl delete secrets spark-webhook-certs -n sparkoper
  ./hack_gen_orig.sh -n sparkoper -s sparkoperator-webhook
• Deleted the below config and pod. Operator pod got re-deployed and webhookconfig got recreated.. and seems to have worked..
  kubectl delete mutatingwebhookconfigurations sparkoperator-webhook-config
  kubectl delete pod/sparkoperator-55b7475b59-tb95m

[spkchels]

@Fixmetal what do you see in the apiserver logs? There could be hints for why the driver is not starting. Also, the Operator pod logs. I saw TLS issues in both these places which was causing the driver to fail.

[Fixmetal]

For reference: there's something odd on the k8s cluster I've been using. We have 3 SCs on the cluster, which are:

• local-path (Rancher fake host past plugin)
• rook-ceph
• trident over Netapp SAN

After a little investigation, I discovered that only local-path would behave correctly. Both the other (real I'd say) 2 SCs would just get a timeout. This has happened in a standard project too which includes a standard Deployment and no CRs.
So in my case, it seems it's not SparkOperator's fault and we need to investigate further about what's going on with the cluster itself. I will update just for reference when we find our root cause.

[glitch-k8s]

@Fixmetal @spkchels @rakshithvasudev in my case volume is getting mounted in driver and failing in executor.

k8s: v1.18.+
Spark Operator: v1beta2-1.2.0-3.0.0 / v1beta2-1.1.1.1-2.4.5
Environment : AWS

When I checked exception logs of spark operator then I got following:

I1214 17:00:00.494155 9 webhook.go:246] Serving admission request
2020/12/14 17:00:00 http: panic serving 10.1.22.119:57744: runtime error: index out of range [1] with length 1
goroutine 149 [running]:
net/http.(conn).serve.func1(0xc000256000)
/usr/local/go/src/net/http/server.go:1772 +0x139
panic(0x137a660, 0xc0006234c0)
/usr/local/go/src/runtime/panic.go:973 +0x396
github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook.addVolumeMount(0xc000a80a80, 0xc000503090, 0xf, 0x0, 0xc000503080, 0xb, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook/patch.go:177 +0x57f
github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook.addVolumes(0xc000a80a80, 0xc0006de000, 0x142ebf9, 0xa, 0xc0002e9ba8)
/go/src/github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook/patch.go:144 +0x570
github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook.patchSparkPod(0xc000a80a80, 0xc0006de000, 0x17, 0xc0006de000, 0x0)
/go/src/github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook/patch.go:52 +0xc5
github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook.mutatePods(0xc0004b13b0, 0x160b060, 0xc00032afe0, 0x7fff47d5ea51, 0xa, 0x160c4e0, 0xc0004b13b0, 0x160c4e0)
/go/src/github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook/webhook.go:554 +0x591
github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook.(WebHook).serve(0xc00047e000, 0x1625460, 0xc000a62700, 0xc0006c2300)
/go/src/github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/webhook/webhook.go:278 +0xb1a
net/http.HandlerFunc.ServeHTTP(0xc00032aff0, 0x1625460, 0xc000a62700, 0xc0006c2300)
/usr/local/go/src/net/http/server.go:2012 +0x44
net/http.(ServeMux).ServeHTTP(0xc0000bce00, 0x1625460, 0xc000a62700, 0xc0006c2300)
/usr/local/go/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0002f0380, 0x1625460, 0xc000a62700, 0xc0006c2300)
/usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(conn).serve(0xc000256000, 0x162b7e0, 0xc0001a3a80)
/usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2933 +0x35c
I1214 17:00:00.505935 9 spark_pod_eventhandler.go:47] Pod test-1607965184791-exec-1 added in namespace aws-test

[sanazba]

@rakshithvasudev
would you please explain how did you solve the problem?

[rakshithvasudev]

@rakshithvasudev
would you please explain how did you solve the problem?

I didn't use spark operator. I ended up using the default spark on k8s. After I moved on, I saw some comments about why this might have happened (having kubeflow on the cluster which comes with a spark operator), I wasn't in a position to get rid of kubeflow from the cluster or it's spark operator. So as it stands today, I'm just using the default spark on k8s.

Thanks