Spark-on-k8s-operator: Unable to pass environment variables in the example app on AKS/EKS 1.19. (works fine on 1.17)

Created on 16 Apr 2021 · 8Comments · Source: GoogleCloudPlatform/spark-on-k8s-operator

  I have installed the spark operator on AKS 1.19 but when I pass the environment variable in the given example 
  https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-py-pi.yaml

  it does not get passed to the pod definition. I have already enabled the web hook as well.

  Below is the pod manifest (spark.-.Copy.-.Copy.log) I want to create.

spark - Copy - Copy.log

Source

Venkatesh89-Github

👍2

Most helpful comment

Hey guys, we were able to fix the issue by using the image with tag as "latest" and enable the webhook with below cleanup annotations.
cleanupAnnotations:
helm.sh/hook: pre-delete
helm.sh/hook-delete-policy: hook-succeeded

Venkatesh89-Github on 29 Apr 2021

👍2

All 8 comments

Having the same issue on EKS 1.18, and 1.17...
I see the values passed in through my helm get values command.

Also if I check my sparkApplication by doing kubectl get sparkapp patiente2e -o yaml, I do see the env variables listed in there, shown here:

$ kg sparkapp -n patiente2e -o  yaml
apiVersion: v1
items:
- apiVersion: sparkoperator.k8s.io/v1beta2
  kind: SparkApplication
  metadata:
    annotations:
      meta.helm.sh/release-name: qa-patiente2e
      meta.helm.sh/release-namespace: patiente2e
    creationTimestamp: "2021-03-26T19:52:24Z"
    generation: 40
    labels:
      app.kubernetes.io/managed-by: Helm
    managedFields:
    - apiVersion: sparkoperator.k8s.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          .: {}
          f:driver:
            .: {}
            f:annotations:
              .: {}
              f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
              f:executionTime: {}
            f:env: {}
            f:labels:
              .: {}
              f:version: {}
            f:memory: {}
            f:serviceAccount: {}
            f:volumeMounts: {}
          f:dynamicAllocation:
            .: {}
            f:enabled: {}
            f:initialExecutors: {}
            f:maxExecutors: {}
            f:minExecutors: {}
          f:executor:
            .: {}
            f:coreLimit: {}
            f:coreRequest: {}
            f:cores: {}
            f:env: {}
            f:envFrom: {}
            f:labels:
              .: {}
              f:version: {}
            f:memory: {}
            f:volumeMounts: {}
          f:image: {}
          f:imagePullPolicy: {}
          f:imagePullSecrets: {}
          f:mainApplicationFile: {}
          f:mainClass: {}
          f:mode: {}
          f:pythonVersion: {}
          f:restartPolicy:
            .: {}
            f:type: {}
          f:sparkConf:
            .: {}
            f:spark.kubernetes.local.dirs.tmpfs: {}
          f:sparkUIOptions:
            .: {}
            f:ingressAnnotations:
              .: {}
              f:cert-manager.io/cluster-issuer: {}
              f:external-dns.alpha.kubernetes.io/target: {}
              f:kubernetes.io/ingress.class: {}
            f:ingressTLS: {}
          f:sparkVersion: {}
          f:type: {}
          f:volumes: {}
      manager: Go-http-client
      operation: Update
      time: "2021-04-20T16:58:54Z"
    - apiVersion: sparkoperator.k8s.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:deps: {}
          f:sparkUIOptions:
            f:servicePort: {}
        f:status:
          .: {}
          f:applicationState:
            .: {}
            f:state: {}
          f:driverInfo:
            .: {}
            f:podName: {}
            f:webUIAddress: {}
            f:webUIIngressAddress: {}
            f:webUIIngressName: {}
            f:webUIPort: {}
            f:webUIServiceName: {}
          f:executionAttempts: {}
          f:executorState:
            .: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-1: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-2: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-3: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-4: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-5: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-6: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-7: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-8: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-9: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-10: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-11: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-12: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-13: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-14: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-15: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-16: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-17: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-18: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-19: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-20: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-21: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-22: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-23: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-24: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-25: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-26: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-27: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-28: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-29: {}
            f:spark4pe2epostvalidator-fb8fe578f038a36d-exec-30: {}
          f:lastSubmissionAttemptTime: {}
          f:sparkApplicationId: {}
          f:submissionAttempts: {}
          f:submissionID: {}
          f:terminationTime: {}
      manager: spark-operator
      operation: Update
      time: "2021-04-20T17:10:20Z"
    name: patiente2e
    namespace: patiente2e
    resourceVersion: "11219748"
    selfLink: /apis/sparkoperator.k8s.io/v1beta2/namespaces/patiente2e/sparkapplications/patiente2e
    uid: bade5903-b4ac-4807-9178-3dbf89796bed
  spec:
    driver:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
        executionTime: "1618937933"
      env:
      - **name: ConcertAI_ENV
        value: UAT**
      labels:
        version: 3.0.0
      memory: 16000m
      serviceAccount: sparkoperator-spark
      volumeMounts:
      - mountPath: /tmp
        name: spark-local-dir
    dynamicAllocation:
      enabled: true
      initialExecutors: 2
      maxExecutors: 50
      minExecutors: 2
    executor:
      coreLimit: 3000m
      coreRequest: 2500m
      cores: 3
      **env:
      - name: ConcertAI_ENV
        value: UAT**
      envFrom:
      - configMapRef:
          name: qa-patiente2e
      labels:
        version: 3.0.0
      memory: 20000m
      volumeMounts:
      - mountPath: /tmp
        name: spark-local-dir

However when I exec into one of the executor pods, or the driver pods, the env variable is not passing in:

# printenv KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 SPARK_EXECUTOR_ID=1 HOSTNAME=spark4pe2epostvalidator-fb8fe578f038a36d-exec-1 PYSPARK_DRIVER_PYTHON=python3 JAVA_HOME=/usr/local/openjdk-8 PATIENTE2E_UI_SVC_PORT=tcp://172.20.132.57:4040 JAVA_BASE_URL=https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jre_ PATIENTE2E_UI_SVC_PORT_4040_TCP_PROTO=tcp PWD=/app/patiente2e-cicd SPARK_JAVA_OPT_0=-Dspark.driver.blockManager.port=7079 SPARK_JAVA_OPT_1=-Dspark.driver.port=7078 JAVA_URL_VERSION=8u252b09 SPARK_EXTRA_CLASSPATH=/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/hadoop/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/opt/hadoop/share/hadoop/tools/lib/*:/hadoop/share/hadoop/tools/lib/*:/opt/spark/jars/* HOME=/root LANG=C.UTF-8 KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443 PATIENTE2E_UI_SVC_SERVICE_HOST=172.20.132.57 PATIENTE2E_UI_SVC_PORT_4040_TCP_ADDR=172.20.132.57 PYSPARK_PYTHON=python3 TERM=xterm HADOOP_HOME=/opt/hadoop SPARK_EXECUTOR_CORES=3 SPARK_APPLICATION_ID=spark-44e0897bd17c4f6cb8dd5cb00153cc5e SPARK_USER=root PATIENTE2E_UI_SVC_PORT_4040_TCP=tcp://172.20.132.57:4040 SPARK_LOCAL_DIRS=/var/data/spark-32f2fdad-4420-46ae-87af-7b209ca3bf14 SHLVL=1 SPARK_HOME=/opt/spark KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1 SPARK_EXECUTOR_MEMORY=20000m LD_LIBRARY_PATH=/lib64 PATIENTE2E_UI_SVC_SERVICE_PORT_SPARK_DRIVER_UI_PORT=4040 SPARK_CONF_DIR=/opt/spark/conf SPARK_DIST_CLASSPATH=/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/hadoop/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/opt/hadoop/share/hadoop/tools/lib/* KUBERNETES_SERVICE_HOST=172.20.0.1 KUBERNETES_PORT=tcp://172.20.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATIENTE2E_UI_SVC_SERVICE_PORT=4040 PATIENTE2E_UI_SVC_PORT_4040_TCP_PORT=4040 PATH=/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SPARK_DRIVER_URL=spark://CoarseGrainedScheduler@patiente2e-d6bf2778f0387b7f-driver-svc.patiente2e.svc:7078 SPARK_EXECUTOR_POD_IP=100.64.10.194 JAVA_VERSION=8u252 _=/usr/bin/printenv

JunaidChaudry on 20 Apr 2021

👍1

@Venkatesh89-Github Can you Update the Title so that it covers both of us? I think it is a sparkOperator issue, rather than AKS/EKS specific?

JunaidChaudry on 20 Apr 2021

👍1

Can someone help on this issue please?

Venkatesh89-Github on 22 Apr 2021

I think I am running into the same issue. In my case, I am using RKE2.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8+rke2r1", GitCommit:"fd5d41537aee486160ad9b5356a9d82363273721", GitTreeState:"clean", BuildDate:"2021-02-26T00:03:32Z", GoVersion:"go1.15.8b5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8+rke2r1", GitCommit:"fd5d41537aee486160ad9b5356a9d82363273721", GitTreeState:"clean", BuildDate:"2021-02-26T00:03:32Z", GoVersion:"go1.15.8b5", Compiler:"gc", Platform:"linux/amd64"}

This works for me:

    envSecretKeyRefs:
      AWS_ACCESS_KEY_ID:
        name: aws-s3-credentials
        key: AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY:
        name: aws-s3-credentials
        key: AWS_SECRET_ACCESS_KEY

However, all of the following do not work:

    env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: aws-s3-credentials
            key: AWS_ACCESS_KEY_ID
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: aws-s3-credentials
            key: AWS_SECRET_ACCESS_KEY

and

    envFrom:
    - secretRef:
        name: aws-s3-credentials

The main concern I have is that envSecretKeyRefs is set to be deprecated, but the official Kubernetes methods for passing in the env vars does not seem to working as expected.

According to the SparkPodSpec, all of these options should work, but they don't (in my testing)...

Edit: It is probably worth noting that I am able to use envFrom in my Kubernetes environment in an nginx deployment, so this problem does seem to be specific to the SparkApplication kind.

swill on 27 Apr 2021

👍1

1224 should be related to this issue.

please check the k8s api server logs

felixng-umh on 29 Apr 2021

Same issues happening on GKE 1.19.8-gke.1600. Envvars are correctly defined in the SparkApplication, but not propagated to the driver and executor pods. Running the latest SparkOperator Helm chart v1.0.9.

MikeBlomm on 29 Apr 2021

Venkatesh89-Github on 29 Apr 2021

👍2

@Venkatesh89-Github Unfortunately this fix did not work for us. Are there perhaps any other configurations to be tweaked?

EDIT
I eventually did get it to work with the mentioned fix by deleting and reinstalling the operator manually with Helm. Also seems to work with image tag v1beta2-1.2.1-3.0.0

MikeBlomm on 4 May 2021

👍1

Was this page helpful?

0 / 5 - 0 ratings