Spark-on-k8s-operator: Mounting Volumes - nothing happens

[michalzxc]

Hi,
I can't make "Volume Mounting" to work, even with the example: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml

kubectl --namespace spark describe pod spark-pi-driver
[...]
Volumes:
  spark-local-dir-1:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  spark-conf-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      spark-pi-1553707634744-driver-conf-map
    Optional:  false
  spark-operator-spark-token-7w84x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  spark-operator-spark-token-7w84x
    Optional:    false

There is no "test-volume" anywhere there

At the same time when I describe sparkapplications I see volume over there:

kubectl --namespace spark describe sparkapplications.sparkoperator.k8s.io spark-pi
[...]
    Volume Mounts:
      Mount Path:  /tmp
      Name:        test-volume
[...]
  Volumes:
    Host Path:
      Path:  /tmp
      Type:  Directory
    Name:    test-volume

[liyinan926]

Do you have the mutating webhook enabled? One thing you want to check is if there's an OwnerReference for the owning SparkApplication on your driver pod. If no, then it's clearly that the webhook is either not enabled or not functioning properly.

[michalzxc]

I installed it with that command: helm install incubator/sparkoperator --namespace spark --set "sparkJobNamespace=spark" --set "enableWebhook=true" --name spark-operator
I can see webhook service in spark namespace

[liyinan926]

Did you check the OwnerReference?

[michalzxc]

Is that this Adnotatipon you are asking about?

~❯ kubectl --namespace spark get pod  spark-pi-driver -o yaml | grep -i ownerreference
    sparkoperator.k8s.io/ownerreference: ChBTcGFya0FwcGxpY2F0aW9uGghzcGFyay1waSIkYjFmOTFiYTQtNTE4MS0xMWU5LWFjYWEtMDY5MDc4NGUxNDNjKhxzcGFya29wZXJhdG9yLms4cy5pby92MWJldGExMAE=

[liyinan926]

No, I meant the ownerReferences field in the pod metadata. BTW: the operator no longer uses the annotation you pasted above. Which version of the operator are you running?

[michalzxc]

I am not seeing this one. This is entire Pod:

```~❯ kubectl --namespace spark get pod spark-pi-driver -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2019-03-29T09:53:17Z"
labels:
spark-app-selector: spark-062c3d6b66f84a77a87bbd2260ef042f
spark-role: driver
sparkoperator.k8s.io/app-name: spark-pi
sparkoperator.k8s.io/launched-by-spark-operator: "true"
sparkoperator.k8s.io/submission-id: spark-pi-1553853194906163458
version: 2.4.0
name: spark-pi-driver
namespace: spark
resourceVersion: "9548018"
selfLink: /api/v1/namespaces/spark/pods/spark-pi-driver
uid: 7a50ef59-5208-11e9-acaa-0690784e143c
spec:
containers:

• args:
  
  
  
  • driver
  
  
  • --properties-file
  
  
  • /opt/spark/conf/spark.properties
  
  
  • --class
  
  
  • org.apache.spark.examples.SparkPi
  
  
  • spark-internal
    
    env:
  
  
  • name: SPARK_DRIVER_BIND_ADDRESS
    
    valueFrom:
    
    fieldRef:
    
    apiVersion: v1
    
    fieldPath: status.podIP
  
  
  • name: SPARK_LOCAL_DIRS
    
    value: /var/data/spark-d8a2bc42-33da-40af-8d96-83ff79026ce6
  
  
  • name: SPARK_CONF_DIR
    
    value: /opt/spark/conf
    
    image: gcr.io/spark-operator/spark:v2.4.0
    
    imagePullPolicy: Always
    
    name: spark-kubernetes-driver
    
    ports:
  
  
  • containerPort: 7078
    
    name: driver-rpc-port
    
    protocol: TCP
  
  
  • containerPort: 7079
    
    name: blockmanager
    
    protocol: TCP
  
  
  • containerPort: 4040
    
    name: spark-ui
    
    protocol: TCP
    
    resources:
    
    limits:
    
    cpu: 200m
    
    memory: 896Mi
    
    requests:
    
    cpu: 100m
    
    memory: 896Mi
    
    terminationMessagePath: /dev/termination-log
    
    terminationMessagePolicy: File
    
    volumeMounts:
  
  
  • mountPath: /var/data/spark-d8a2bc42-33da-40af-8d96-83ff79026ce6
    
    name: spark-local-dir-1
  
  
  • mountPath: /opt/spark/conf
    
    name: spark-conf-volume
  
  
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    
    name: spark-operator-spark-token-m28hv
    
    readOnly: true
    
    dnsPolicy: ClusterFirst
    
    nodeName: ip-172-17-10-13.eu-west-1.compute.internal
    
    priority: 0
    
    restartPolicy: Never
    
    schedulerName: default-scheduler
    
    securityContext: {}
    
    serviceAccount: spark-operator-spark
    
    serviceAccountName: spark-operator-spark
    
    terminationGracePeriodSeconds: 30
    
    tolerations:
  
  
• effect: NoExecute
  key: node.kubernetes.io/not-ready
  operator: Exists
  tolerationSeconds: 300
• effect: NoExecute
  key: node.kubernetes.io/unreachable
  operator: Exists
  tolerationSeconds: 300
  volumes:
• emptyDir: {}
  name: spark-local-dir-1
• configMap:
  defaultMode: 420
  name: spark-pi-1553853196427-driver-conf-map
  name: spark-conf-volume
• name: spark-operator-spark-token-m28hv
  secret:
  defaultMode: 420
  secretName: spark-operator-spark-token-m28hv
  status:
  conditions:
• lastProbeTime: null
  lastTransitionTime: "2019-03-29T09:53:17Z"
  status: "True"
  type: Initialized
• lastProbeTime: null
  lastTransitionTime: "2019-03-29T09:53:20Z"
  status: "True"
  type: Ready
• lastProbeTime: null
  lastTransitionTime: null
  status: "True"
  type: ContainersReady
• lastProbeTime: null
  lastTransitionTime: "2019-03-29T09:53:17Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
• containerID: docker://58733f0cd88c983d238dba2a8ba05387202dadbf0acf80dc3d32717a6e0c4357
  image: gcr.io/spark-operator/spark:v2.4.0
  imageID: docker-pullable://gcr.io/spark-operator/spark@sha256:375cd079d14179687b37625ce850c905c452010325d62c35618b8e0c8ddc749b
  lastState: {}
  name: spark-kubernetes-driver
  ready: true
  restartCount: 0
  state:
  running:
  startedAt: "2019-03-29T09:53:19Z"
  hostIP: 172.17.10.13
  phase: Running
  podIP: 10.35.0.1
  qosClass: Burstable
  startTime: "2019-03-29T09:53:17Z"

Describe:
```~❯ kubectl --namespace spark describe pod spark-pi-driver
Name:               spark-pi-driver
Namespace:          spark
Priority:           0
PriorityClassName:  <none>
Node:               ip-172-17-10-13.eu-west-1.compute.internal/172.17.10.13
Start Time:         Fri, 29 Mar 2019 09:53:17 +0000
Labels:             spark-app-selector=spark-062c3d6b66f84a77a87bbd2260ef042f
                    spark-role=driver
                    sparkoperator.k8s.io/app-name=spark-pi
                    sparkoperator.k8s.io/launched-by-spark-operator=true
                    sparkoperator.k8s.io/submission-id=spark-pi-1553853194906163458
                    version=2.4.0
Annotations:        <none>
Status:             Succeeded
IP:                 10.35.0.1
Containers:
  spark-kubernetes-driver:
    Container ID:  docker://58733f0cd88c983d238dba2a8ba05387202dadbf0acf80dc3d32717a6e0c4357
    Image:         gcr.io/spark-operator/spark:v2.4.0
    Image ID:      docker-pullable://gcr.io/spark-operator/spark@sha256:375cd079d14179687b37625ce850c905c452010325d62c35618b8e0c8ddc749b
    Ports:         7078/TCP, 7079/TCP, 4040/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      driver
      --properties-file
      /opt/spark/conf/spark.properties
      --class
      org.apache.spark.examples.SparkPi
      spark-internal
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 29 Mar 2019 09:53:19 +0000
      Finished:     Fri, 29 Mar 2019 09:54:45 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  896Mi
    Requests:
      cpu:     100m
      memory:  896Mi
    Environment:
      SPARK_DRIVER_BIND_ADDRESS:   (v1:status.podIP)
      SPARK_LOCAL_DIRS:           /var/data/spark-d8a2bc42-33da-40af-8d96-83ff79026ce6
      SPARK_CONF_DIR:             /opt/spark/conf
    Mounts:
      /opt/spark/conf from spark-conf-volume (rw)
      /var/data/spark-d8a2bc42-33da-40af-8d96-83ff79026ce6 from spark-local-dir-1 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from spark-operator-spark-token-m28hv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  spark-local-dir-1:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  spark-conf-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      spark-pi-1553853196427-driver-conf-map
    Optional:  false
  spark-operator-spark-token-m28hv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  spark-operator-spark-token-m28hv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age    From                                                 Message
  ----     ------       ----   ----                                                 -------
  Normal   Scheduled    3m44s  default-scheduler                                    Successfully assigned spark/spark-pi-driver to ip-172-17-10-13.eu-west-1.compute.internal
  Warning  FailedMount  3m44s  kubelet, ip-172-17-10-13.eu-west-1.compute.internal  MountVolume.SetUp failed for volume "spark-conf-volume" : configmaps "spark-pi-1553853196427-driver-conf-map" not found
  Normal   Pulling      3m42s  kubelet, ip-172-17-10-13.eu-west-1.compute.internal  pulling image "gcr.io/spark-operator/spark:v2.4.0"
  Normal   Pulled       3m42s  kubelet, ip-172-17-10-13.eu-west-1.compute.internal  Successfully pulled image "gcr.io/spark-operator/spark:v2.4.0"
  Normal   Created      3m42s  kubelet, ip-172-17-10-13.eu-west-1.compute.internal  Created container
  Normal   Started      3m42s  kubelet, ip-172-17-10-13.eu-west-1.compute.internal  Started container

I made sure to upgrade version to newest one:
~❯ helm ls|grep spark spark-operator 1 Fri Mar 29 09:52:27 2019 DEPLOYED sparkoperator-0.1.13 v1beta1-0.8.1-2.4.0 spark

This is Amazon EKS cluster, version: v1.11.5

[michalzxc]

Amazon is claiming that EKS should support Dynamic Admission Webhooks https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-eks-enables-support-for-kubernetes-dynamic-admission-cont/

[liyinan926]

I don't see the OwnerReference in the resource metadata. Apparently the webhook didn't work properly.

[yuchaoran2011]

@michalzxc Have you tried this example? Note that you need to first create a ConfigMap called dummy-cm in the job namespace before running the example. I ask because some Kubernetes distributions (not sure about Amazon EKS specifically) require special settings to enable hostPath mounting, which is what the spark-pi example is doing. But the example I referred to should work without special setup, as long as the webhook is enabled.

[michalzxc]

@yuchaoran2011 I tried also using existing PVC targeting NFS storage, but claim isn't mounted:
```
~❯ kubectl --namespace spark describe pvc spark-checkpoint
Name: spark-checkpoint
Namespace: spark
StorageClass: efs
Status: Bound
Volume: pvc-cd4742f9-50a0-11e9-acaa-0690784e143c
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-class: efs
volume.beta.kubernetes.io/storage-provisioner: stg01.tixey.com/aws-efs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWX
Events:
Mounted By:

And PV:

~❯ kubectl --namespace spark describe pv pvc-cd4742f9-50a0-11e9-acaa-0690784e143c
Name: pvc-cd4742f9-50a0-11e9-acaa-0690784e143c
Labels:
Annotations: pv.beta.kubernetes.io/gid: 40001
pv.kubernetes.io/provisioned-by: stg01.tixey.com/aws-efs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: efs
Status: Bound
Claim: spark/spark-checkpoint
Reclaim Policy: Delete
Access Modes: RWX
Capacity: 10Gi
Node Affinity:
Message:
Source:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: fs-8a751542.efs.eu-west-1.amazonaws.com
Path: /spark-checkpoint-pvc-cd4742f9-50a0-11e9-acaa-0690784e143c
ReadOnly: false
Events:
```

[liyinan926]

Not sure if you have found the cause of the issue and resolved it. Feel free to reopen.

[Jeffwan]

I confirmed it's working on EKS. For people may have similar issue, please follow steps.

Make sure you enable webhook during installation.

helm install incubator/sparkoperator --namespace spark-operator --set enableWebhook=true

Confirm driver pods has ownerRerence:

k get pods wordcount-driver -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2019-08-28T22:04:16Z"
  labels:
    spark-app-selector: spark-75ef014093894ac885f44cf7ca53a389
    spark-role: driver
    sparkoperator.k8s.io/app-name: wordcount
    sparkoperator.k8s.io/launched-by-spark-operator: "true"
    sparkoperator.k8s.io/submission-id: 6099674c-f751-4e16-8dfa-700d7e8fcb69
    version: 2.4.3
  name: wordcount-driver
  namespace: default
  ownerReferences:
  - apiVersion: sparkoperator.k8s.io/v1beta1
    controller: true
    kind: SparkApplication
    name: wordcount
    uid: c531caec-c9df-11e9-ae58-06d53bb9195a
  resourceVersion: "1010596"
  selfLink: /api/v1/namespaces/default/pods/wor

You can also see driver pod has volume mounted.

  volumes:
  - emptyDir: {}
    name: spark-local-dir-1
  - configMap:
      defaultMode: 420
      name: wordcount-1567029854949-driver-conf-map
    name: spark-conf-volume
  - name: spark-token-nnjsb
    secret:
      defaultMode: 420
      secretName: spark-token-nnjsb
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim

Here's an entire job example.

apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: wordcount
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: "seedjeffwan/spark:2.4.3"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.JavaWordCount
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar"
  arguments:
    - /data/out.txt
  sparkVersion: "2.4.3"
  restartPolicy:
    type: Never
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim
  driver:
    cores: 0.1
    coreLimit: "200m"
    memory: "512m"
    labels:
      version: 2.4.3
    serviceAccount: spark
    volumeMounts:
      - name: persistent-storage
        mountPath: /data
  executor:
    cores: 1
    instances: 1
    memory: "1g"
    labels:
      version: 2.4.3
    volumeMounts:
      - name: persistent-storage
        mountPath: /data

[rakshithvasudev]

I've installed the webhook, but I don't see the ownerReferences field. Does anybody have the same issue? Here's how I installed the operator:
helm install spark-operator incubator/sparkoperator --namespace spark-rak --set sparkJobNamespace=spark-rak,enableWebhook=true,operatorVersion=v1beta2-1.1.2-2.4.5

I'm also not able to mount, more here: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/946

[unnikpr]

Hi,
I have the same issue with kubernees 1.20 and spark on the 3 node cluster. enabled webhook and running the same. but the mount is not working as expected. I don't see the ownerReferences, How to fix this issue? Installe dusing
helm install hmtx-spark-operator spark-operator/spark-operator --namespace spark-operator --create-namespace --set enableWebhook=true --set webhookPort=443 --set serviceAccounts.spark.name=spark

[dvaldivia]

if I enable the webhooks I can see the secret being generated, but still no volume gets mounted

[kzfile]

if I enable the webhooks I can see the secret being generated, but still no volume gets mounted

I failed on spark-opearator tag "v1beta2-1.2.0-3.0.0" but success on tag "latest".

[michalzxc]

Hi,
Yes it was resolved for me, in my case problem was completely somewhere else. I was using weave with eks (what is more stable in my opinion than aws-cni), but the thing what I didn't know is that using different CNI with eks breaks MuttatingAdmissionWebhooks all together. So as result masters (managed as a "service" without weave pod/networking) wasn't able to reach operator so mutation never happened. Spark was my first thing using Muttations in kubernetes, so I had no idea at a time that custom CNI could be at fault

[unnikpr]

Hi,

Thanks for the update. I was able to fix this by reinstalling the webhook
operator and able to mount the volumes.

Regards,
Unni.

On Thu, Apr 22, 2021 at 9:03 PM michalzxc @.*> wrote:

Hi,
Yes it was resolved for me, in my case problem was completely somewhere
else. I was using weave with eks (what is more stable in my opinion than
aws-cni), but the thing what I didn't know is that using different CNI with
eks breaks MuttatingAdmissionWebhooks all together. So as result masters
(managed as a "service" without weave pod/networking) wasn't able to reach
operator so mutation never happened. Spark was my first thing using
Muttations in kubernetes, so I had no idea at a time that weave could be at
fault

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/455#issuecomment-824946254,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOCLKBSBJPZHJ5VV7GOOPDTKA6SZANCNFSM4HBZUOLQ
.

[ziaulhaqueansari]

Hi, Thanks for the update. I was able to fix this by reinstalling the webhook operator and able to mount the volumes. Regards, Unni.

If you don't mind, can you please tell the steps you took to re-install webhook operator?
Anybody facing same issue on AKS? I have tried to re-install multiple times on AKS with webhook enabled, it still does not work. I see the webhook getting creating. This was working and then stopped suddenly.

Thanks in advance!!!
With Regards,
Zia

[unnikpr]

Hi,
The following one is I executed

curl -fsSL -o get_helm.sh
https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

chmod 700 get_helm.sh

/get_helm.sh

Ref: https://helm.sh/docs/intro/install/
Configure the operator POD with web hook

helm repo add spark-operator
https://googlecloudplatform.github.io/spark-on-k8s-operator
helm install my-release spark-operator/spark-operator --namespace
spark-operator --create-namespace --
set webhook.enable=true

Regards,
Unni

On Mon, Apr 26, 2021, 8:28 PM ziaulhaqueansari @.*>
wrote:

Hi, Thanks for the update. I was able to fix this by reinstalling the
webhook operator and able to mount the volumes. Regards, Unni.

If you don't mind, can you please tell the steps you took to re-install
webhook operator?
Anybody facing same issue on AKS? I have tried to re-install multiple
times on AKS with webhook enabled, it still does not work. I see the
webhook getting creating. This was working and then stopped suddenly.

Thanks in advance!!!
With Regards,
Zia

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/455#issuecomment-826903729,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABOCLKHKMLGSH6SMRF7FXFLTKV5QZANCNFSM4HBZUOLQ
.

[joanjiao2016]

I change the volume name to spark-local-dir-1 , the pvc mounted