Spark-on-k8s-operator: Environmental variables are not available when using `.spec.driver.env` or `.spec.driver.envFrom`

Created on 25 Feb 2020  Â·  2Comments  Â·  Source: GoogleCloudPlatform/spark-on-k8s-operator

Hi guys,

I have a problem with getting Environmental Variables in my Python script (pyspark) when using:

  • .spec.driver.env (and .spec.executor.env)
  • .spec.driver.envFrom (and .spec.executor.envFrom)

It only works, when i use .spec.driver.envVars, which seems to be deprecated according to Specifying Environment Variables.

Can you please let me know if there is something what I am doing wrong, or is there a bug in Spark Operator?

To demonstrate this issue, I have written short script, which just checks existence of env variables:

test.py

import os

if __name__ == "__main__":

    variables = ['START_TIME', 'END_TIME', 'STEP_SIZE', 'WINDOW_SIZE', 'BATCH_SIZE', 'HOME']

    for v in variables:
        if v in os.environ:
            print(f"Env variable {v} exists. ({v}={os.environ[v]})")
        else:
            print(f"Env variable {v} doesn't exists")

I am using spark operator in version: v1beta2-1.1.0-2.4.5 deployed from helm3:

config.yaml

operatorImageName: gcr.io/spark-operator/spark-operator
operatorVersion: v1beta2-1.1.0-2.4.5

sparkJobNamespace: "spark"

nodeSelector:
  cloud.google.com/gke-nodepool: spark-pool

serviceAccounts:
  spark:
    name: spark

deploy.sh

#!/bin/bash

RELEASENAME="sparkoperator"
NAMESPACE="spark-operator"
JOBSNAMESPACE="spark"

kubectl create namespace ${NAMESPACE}
kubectl create namespace ${JOBSNAMESPACE}
helm upgrade --install --values ./config.yaml ${RELEASENAME} incubator/sparkoperator --namespace ${NAMESPACE}
  1. Deploy SparkApplication using: .spec.driver.env (and .spec.executor.env)

Expected result: application test.py should print environmental variables

test-env.yml

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: counter-example
  namespace: spark
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "gcr.io/pathr-operations/spark-jobs:v0.0.14"
  imagePullPolicy: Always
  mainApplicationFile: local:///jobs/count-points/test.py
  sparkVersion: "2.4.5"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 2.4.5
    serviceAccount: spark
    env:
      - name: START_TIME
        value: "1583121125371"
      - name: END_TIME
        value: "1583124125371"
      - name: STEP_SIZE
        value: "60000"
      - name: WINDOW_SIZE
        value: "60000"
      - name: BATCH_SIZE
        value: "10"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.5
    env:
      - name: START_TIME
        value: "1583121125371"
      - name: END_TIME
        value: "1583124125371"
      - name: STEP_SIZE
        value: "60000"
      - name: WINDOW_SIZE
        value: "60000"
      - name: BATCH_SIZE
        value: "10"

Deploy SparkApplication:

# kubectl apply -f test-env.yml
sparkapplication.sparkoperator.k8s.io/test-env created

Log from driver (snippet):

Env variable START_TIME doesn't exists
Env variable END_TIME doesn't exists
Env variable STEP_SIZE doesn't exists
Env variable WINDOW_SIZE doesn't exists
Env variable BATCH_SIZE doesn't exists
Env variable HOME exists. (HOME=/root)

Result: NONE of variables specified in .spec.driver.env is visible to driver.

Log from spark operator (snippet):

I0225 20:54:04.895552       9 controller.go:179] SparkApplication spark/test-env was added, enqueueing it for submission
I0225 20:54:04.895738       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:04.895886       9 submission.go:65] spark-submit arguments: [/opt/spark/bin/spark-submit --master k8s://https://10.20.0.1:443 --deploy-mode cluster --conf spark.kubernetes.namespace=spark --conf spark.app.name=test-env --conf spark.kubernetes.driver.pod.name=test-env-driver --conf spark.kubernetes.container.image=gcr.io/pathr-operations/spark-jobs:v0.0.14 --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=test-env --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=ce257a99-f8da-49c9-9e64-b6c56f3691e0 --conf spark.driver.cores=1 --conf spark.kubernetes.driver.limit.cores=1200m --conf spark.driver.memory=512m --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.driver.label.version=2.4.5 --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=test-env --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=ce257a99-f8da-49c9-9e64-b6c56f3691e0 --conf spark.executor.instances=1 --conf spark.executor.cores=1 --conf spark.executor.memory=512m --conf spark.kubernetes.executor.label.version=2.4.5 local:///jobs/count-points/test.py]
I0225 20:54:04.895979       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-env", UID:"a29de05f-ad49-463a-9fbb-30973341aa0f", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8964517", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationAdded' SparkApplication test-env was added, enqueuing it for submission
I0225 20:54:06.860222       9 spark_pod_eventhandler.go:47] Pod test-env-driver added in namespace spark.
I0225 20:54:06.868710       9 spark_pod_eventhandler.go:58] Pod test-env-driver updated in namespace spark.
I0225 20:54:06.900590       9 spark_pod_eventhandler.go:58] Pod test-env-driver updated in namespace spark.
I0225 20:54:07.332493       9 controller.go:674] SparkApplication spark/test-env has been submitted
I0225 20:54:07.332572       9 sparkui.go:131] Creating a service test-env-ui-svc for the Spark UI for application test-env
I0225 20:54:07.332736       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-env", UID:"a29de05f-ad49-463a-9fbb-30973341aa0f", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8964517", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSubmitted' SparkApplication test-env was submitted successfully
I0225 20:54:07.350376       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{  0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } { } map[] 0 0}] to [{ ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07.332514559 +0000 UTC m=+43791.749357750 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}]
I0225 20:54:07.360258       9 controller.go:218] SparkApplication spark/test-env was updated, enqueueing it
I0225 20:54:07.362307       9 sparkapp_metrics.go:144] Exporting metrics for test-env; old status: {  0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } { } map[] 0 0} new status: { ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}
I0225 20:54:07.362355       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:07.362384       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:07.362591       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{ ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}]
I0225 20:54:07.369184       9 controller.go:218] SparkApplication spark/test-env was updated, enqueueing it
I0225 20:54:07.370219       9 sparkapp_metrics.go:144] Exporting metrics for test-env; old status: { ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1} new status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}
I0225 20:54:07.370259       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:07.370280       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:07.370343       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}]
I0225 20:54:07.370444       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:08.162165       9 spark_pod_eventhandler.go:58] Pod test-env-driver updated in namespace spark.
I0225 20:54:08.162190       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-env for app update processing.
I0225 20:54:08.162218       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:08.162320       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}]
I0225 20:54:08.162514       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:09.276040       9 spark_pod_eventhandler.go:58] Pod test-env-driver updated in namespace spark.
I0225 20:54:09.276070       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-env for app update processing.
I0225 20:54:09.276104       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:09.276238       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {RUNNING } map[] 1 1}]
I0225 20:54:09.276370       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-env", UID:"a29de05f-ad49-463a-9fbb-30973341aa0f", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8964541", FieldPath:""}): type: 'Normal' reason: 'SparkDriverRunning' Driver test-env-driver is running
I0225 20:54:09.294250       9 controller.go:218] SparkApplication spark/test-env was updated, enqueueing it
I0225 20:54:09.298306       9 sparkapp_metrics.go:144] Exporting metrics for test-env; old status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUBMITTED } map[] 1 1} new status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {RUNNING } map[] 1 1}
I0225 20:54:09.298368       9 metrics.go:111] Incrementing spark_app_running_count with labels map[app_type:Unknown]
I0225 20:54:09.298386       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:09.298410       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:09.298492       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {RUNNING } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {RUNNING } map[] 1 1}]
I0225 20:54:09.298649       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:12.322774       9 spark_pod_eventhandler.go:58] Pod test-env-driver updated in namespace spark.
I0225 20:54:12.322805       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-env for app update processing.
I0225 20:54:12.322837       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:12.322968       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {RUNNING } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12.322861636 +0000 UTC m=+43796.739704840 {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUCCEEDING } map[] 1 1}]
I0225 20:54:12.323058       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-env", UID:"a29de05f-ad49-463a-9fbb-30973341aa0f", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8964552", FieldPath:""}): type: 'Normal' reason: 'SparkDriverCompleted' Driver test-env-driver completed
I0225 20:54:12.354235       9 controller.go:218] SparkApplication spark/test-env was updated, enqueueing it
I0225 20:54:12.365817       9 sparkapp_metrics.go:144] Exporting metrics for test-env; old status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {RUNNING } map[] 1 1} new status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUCCEEDING } map[] 1 1}
I0225 20:54:12.365881       9 metrics.go:124] Decrementing spark_app_running_count with labels map[app_type:Unknown] metricVal to 0
I0225 20:54:12.365898       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:12.365910       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:12.365943       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUCCEEDING } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {COMPLETED } map[] 1 1}]
I0225 20:54:12.366049       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-env", UID:"a29de05f-ad49-463a-9fbb-30973341aa0f", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8964571", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationCompleted' SparkApplication test-env completed
I0225 20:54:12.404210       9 controller.go:218] SparkApplication spark/test-env was updated, enqueueing it
I0225 20:54:12.416022       9 sparkapp_metrics.go:144] Exporting metrics for test-env; old status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {SUCCEEDING } map[] 1 1} new status: {spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {COMPLETED } map[] 1 1}
I0225 20:54:12.416062       9 controller.go:265] Ending processing key: "spark/test-env"
I0225 20:54:12.416084       9 controller.go:258] Starting processing key: "spark/test-env"
I0225 20:54:12.416106       9 controller.go:591] Trying to update SparkApplication spark/test-env, from: [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {COMPLETED } map[] 1 1}] to [{spark-a6b816dc56fe47eba92ae94cadc6a351 ce257a99-f8da-49c9-9e64-b6c56f3691e0 2020-02-25 20:54:07 +0000 UTC 2020-02-25 20:54:12 +0000 UTC {test-env-ui-svc 4040 10.20.14.199:4040   test-env-driver} {COMPLETED } map[] 1 1}]
I0225 20:54:12.416212       9 controller.go:265] Ending processing key: "spark/test-env"
  1. Deploy SparkApplication using: .spec.driver.envFrom (and .spec.executor.envFrom)

test-envfrom.yml

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: test-envfrom
  namespace: spark
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "gcr.io/pathr-operations/spark-jobs:v0.0.14"
  imagePullPolicy: Always
  mainApplicationFile: local:///jobs/count-points/test.py
  sparkVersion: "2.4.5"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 2.4.5
    serviceAccount: spark
    envFrom:
      - configMapRef:
          name: test-config
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.5
    envFrom:
      - configMapRef:
          name: test-config


✗ kubectl apply -f ./test-envfrom.yml
sparkapplication.sparkoperator.k8s.io/test-envfrom created

Log from driver (snippet):

Env variable START_TIME doesn't exists
Env variable END_TIME doesn't exists
Env variable STEP_SIZE doesn't exists
Env variable WINDOW_SIZE doesn't exists
Env variable BATCH_SIZE doesn't exists
Env variable HOME exists. (HOME=/root)

Result: NONE of variables specified in .spec.driver.env is visible to driver.

Log from spark operator (snippet):

I0225 20:55:55.651107       9 controller.go:179] SparkApplication spark/test-envfrom was added, enqueueing it for submission
I0225 20:55:55.651155       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:55:55.651278       9 submission.go:65] spark-submit arguments: [/opt/spark/bin/spark-submit --master k8s://https://10.20.0.1:443 --deploy-mode cluster --conf spark.kubernetes.namespace=spark --conf spark.app.name=test-envfrom --conf spark.kubernetes.driver.pod.name=test-envfrom-driver --conf spark.kubernetes.container.image=gcr.io/pathr-operations/spark-jobs:v0.0.14 --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=test-envfrom --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=14d7958b-d0e9-4dbe-9046-c1a91606793e --conf spark.driver.cores=1 --conf spark.kubernetes.driver.limit.cores=1200m --conf spark.driver.memory=512m --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.driver.label.version=2.4.5 --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=test-envfrom --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=14d7958b-d0e9-4dbe-9046-c1a91606793e --conf spark.executor.instances=1 --conf spark.executor.cores=1 --conf spark.executor.memory=512m --conf spark.kubernetes.executor.label.version=2.4.5 local:///jobs/count-points/test.py]
I0225 20:55:55.651365       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envfrom", UID:"2b9ab898-60e9-4788-89c9-587f94428edc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965046", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationAdded' SparkApplication test-envfrom was added, enqueuing it for submission
I0225 20:55:57.696110       9 spark_pod_eventhandler.go:47] Pod test-envfrom-driver added in namespace spark.
I0225 20:55:57.713366       9 spark_pod_eventhandler.go:58] Pod test-envfrom-driver updated in namespace spark.
I0225 20:55:57.737740       9 spark_pod_eventhandler.go:58] Pod test-envfrom-driver updated in namespace spark.
I0225 20:55:58.033188       9 controller.go:674] SparkApplication spark/test-envfrom has been submitted
I0225 20:55:58.033260       9 sparkui.go:131] Creating a service test-envfrom-ui-svc for the Spark UI for application test-envfrom
I0225 20:55:58.033396       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envfrom", UID:"2b9ab898-60e9-4788-89c9-587f94428edc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965046", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSubmitted' SparkApplication test-envfrom was submitted successfully
I0225 20:55:58.056170       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{  0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } { } map[] 0 0}] to [{ 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58.033208807 +0000 UTC m=+43902.450052029 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}]
I0225 20:55:58.062850       9 controller.go:218] SparkApplication spark/test-envfrom was updated, enqueueing it
I0225 20:55:58.063821       9 sparkapp_metrics.go:144] Exporting metrics for test-envfrom; old status: {  0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } { } map[] 0 0} new status: { 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}
I0225 20:55:58.063867       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:55:58.063891       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:55:58.063957       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{ 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}]
I0225 20:55:58.071664       9 controller.go:218] SparkApplication spark/test-envfrom was updated, enqueueing it
I0225 20:55:58.073542       9 sparkapp_metrics.go:144] Exporting metrics for test-envfrom; old status: { 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1} new status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}
I0225 20:55:58.073577       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:55:58.073597       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:55:58.073646       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}]
I0225 20:55:58.073760       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:55:59.027933       9 spark_pod_eventhandler.go:58] Pod test-envfrom-driver updated in namespace spark.
I0225 20:55:59.027957       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-envfrom for app update processing.
I0225 20:55:59.027976       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:55:59.028023       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}]
I0225 20:55:59.028147       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:56:00.170029       9 spark_pod_eventhandler.go:58] Pod test-envfrom-driver updated in namespace spark.
I0225 20:56:00.170055       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-envfrom for app update processing.
I0225 20:56:00.170079       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:56:00.170190       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {RUNNING } map[] 1 1}]
I0225 20:56:00.170308       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envfrom", UID:"2b9ab898-60e9-4788-89c9-587f94428edc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965071", FieldPath:""}): type: 'Normal' reason: 'SparkDriverRunning' Driver test-envfrom-driver is running
I0225 20:56:00.184405       9 controller.go:218] SparkApplication spark/test-envfrom was updated, enqueueing it
I0225 20:56:00.186899       9 sparkapp_metrics.go:144] Exporting metrics for test-envfrom; old status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUBMITTED } map[] 1 1} new status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {RUNNING } map[] 1 1}
I0225 20:56:00.186945       9 metrics.go:111] Incrementing spark_app_running_count with labels map[app_type:Unknown]
I0225 20:56:00.186956       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:56:00.186971       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:56:00.187015       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {RUNNING } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {RUNNING } map[] 1 1}]
I0225 20:56:00.187118       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:56:03.201064       9 spark_pod_eventhandler.go:58] Pod test-envfrom-driver updated in namespace spark.
I0225 20:56:03.201090       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-envfrom for app update processing.
I0225 20:56:03.201115       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:56:03.201214       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {RUNNING } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03.201139511 +0000 UTC m=+43907.617982717 {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUCCEEDING } map[] 1 1}]
I0225 20:56:03.201286       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envfrom", UID:"2b9ab898-60e9-4788-89c9-587f94428edc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965084", FieldPath:""}): type: 'Normal' reason: 'SparkDriverCompleted' Driver test-envfrom-driver completed
I0225 20:56:03.218721       9 sparkapp_metrics.go:144] Exporting metrics for test-envfrom; old status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {RUNNING } map[] 1 1} new status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUCCEEDING } map[] 1 1}
I0225 20:56:03.218792       9 metrics.go:124] Decrementing spark_app_running_count with labels map[app_type:Unknown] metricVal to 0
I0225 20:56:03.218811       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:56:03.218949       9 controller.go:218] SparkApplication spark/test-envfrom was updated, enqueueing it
I0225 20:56:03.218999       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:56:03.219049       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUCCEEDING } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {COMPLETED } map[] 1 1}]
I0225 20:56:03.219152       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envfrom", UID:"2b9ab898-60e9-4788-89c9-587f94428edc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965099", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationCompleted' SparkApplication test-envfrom completed
I0225 20:56:03.233784       9 controller.go:218] SparkApplication spark/test-envfrom was updated, enqueueing it
I0225 20:56:03.235630       9 sparkapp_metrics.go:144] Exporting metrics for test-envfrom; old status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {SUCCEEDING } map[] 1 1} new status: {spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {COMPLETED } map[] 1 1}
I0225 20:56:03.235676       9 controller.go:265] Ending processing key: "spark/test-envfrom"
I0225 20:56:03.235726       9 controller.go:258] Starting processing key: "spark/test-envfrom"
I0225 20:56:03.235794       9 controller.go:591] Trying to update SparkApplication spark/test-envfrom, from: [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {COMPLETED } map[] 1 1}] to [{spark-3e80f78d97f943b88e81460802019721 14d7958b-d0e9-4dbe-9046-c1a91606793e 2020-02-25 20:55:58 +0000 UTC 2020-02-25 20:56:03 +0000 UTC {test-envfrom-ui-svc 4040 10.20.7.239:4040   test-envfrom-driver} {COMPLETED } map[] 1 1}]
I0225 20:56:03.235901       9 controller.go:265] Ending processing key: "spark/test-envfrom"
  1. Deploy SparkApplication using: .spec.driver.envVars (and .spec.executor.envVars)

test-envvars.yml

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: test-envvars
  namespace: spark
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "gcr.io/pathr-operations/spark-jobs:v0.0.14"
  imagePullPolicy: Always
  mainApplicationFile: local:///jobs/count-points/test.py
  sparkVersion: "2.4.5"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 2.4.5
    serviceAccount: spark
    envVars:
      START_TIME: "1583121125371"
      END_TIME: "1583124125371"
      STEP_SIZE:  "60000"
      WINDOW_SIZE: "60000"
      BATCH_SIZE: "10"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.5
    envVars:
      START_TIME: "1583121125371"
      END_TIME: "1583124125371"
      STEP_SIZE:  "60000"
      WINDOW_SIZE: "60000"
      BATCH_SIZE: "10"


✗ kubectl apply -f ./test-envvars.yml
sparkapplication.sparkoperator.k8s.io/test-envvars created

Log from driver (snippet):

Env variable START_TIME exists. (START_TIME=1583121125371)
Env variable END_TIME exists. (END_TIME=1583124125371)
Env variable STEP_SIZE exists. (STEP_SIZE=60000)
Env variable WINDOW_SIZE exists. (WINDOW_SIZE=60000)
Env variable BATCH_SIZE exists. (BATCH_SIZE=10)
Env variable HOME exists. (HOME=/root)

Result: Variables specified in .spec.driver.env are visible to driver.

Log from spark operator (snippet):

I0225 20:57:19.900816       9 controller.go:179] SparkApplication spark/test-envvars was added, enqueueing it for submission
I0225 20:57:19.900892       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:19.901033       9 submission.go:65] spark-submit arguments: [/opt/spark/bin/spark-submit --master k8s://https://10.20.0.1:443 --deploy-mode cluster --conf spark.kubernetes.namespace=spark --conf spark.app.name=test-envvars --conf spark.kubernetes.driver.pod.name=test-envvars-driver --conf spark.kubernetes.container.image=gcr.io/pathr-operations/spark-jobs:v0.0.14 --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=test-envvars --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=2fc49748-4ecb-4935-aa4f-4cb66d8fc276 --conf spark.driver.cores=1 --conf spark.kubernetes.driver.limit.cores=1200m --conf spark.driver.memory=512m --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.driver.label.version=2.4.5 --conf spark.kubernetes.driverEnv.END_TIME=1583124125371 --conf spark.kubernetes.driverEnv.START_TIME=1583121125371 --conf spark.kubernetes.driverEnv.STEP_SIZE=60000 --conf spark.kubernetes.driverEnv.WINDOW_SIZE=60000 --conf spark.kubernetes.driverEnv.BATCH_SIZE=10 --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=test-envvars --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=2fc49748-4ecb-4935-aa4f-4cb66d8fc276 --conf spark.executor.instances=1 --conf spark.executor.cores=1 --conf spark.executor.memory=512m --conf spark.kubernetes.executor.label.version=2.4.5 --conf spark.executorEnv.WINDOW_SIZE=60000 --conf spark.executorEnv.BATCH_SIZE=10 --conf spark.executorEnv.END_TIME=1583124125371 --conf spark.executorEnv.START_TIME=1583121125371 --conf spark.executorEnv.STEP_SIZE=60000 local:///jobs/count-points/test.py]
I0225 20:57:19.901126       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envvars", UID:"431a855d-e22b-4fee-a600-ac549e498fdc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965454", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationAdded' SparkApplication test-envvars was added, enqueuing it for submission
I0225 20:57:21.880804       9 spark_pod_eventhandler.go:47] Pod test-envvars-driver added in namespace spark.
I0225 20:57:21.895808       9 spark_pod_eventhandler.go:58] Pod test-envvars-driver updated in namespace spark.
I0225 20:57:21.923906       9 spark_pod_eventhandler.go:58] Pod test-envvars-driver updated in namespace spark.
I0225 20:57:22.224156       9 controller.go:674] SparkApplication spark/test-envvars has been submitted
I0225 20:57:22.224235       9 sparkui.go:131] Creating a service test-envvars-ui-svc for the Spark UI for application test-envvars
I0225 20:57:22.224379       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envvars", UID:"431a855d-e22b-4fee-a600-ac549e498fdc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965454", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSubmitted' SparkApplication test-envvars was submitted successfully
I0225 20:57:22.251110       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{  0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } { } map[] 0 0}] to [{ 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22.224180468 +0000 UTC m=+43986.641023659 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}]
I0225 20:57:22.262597       9 controller.go:218] SparkApplication spark/test-envvars was updated, enqueueing it
I0225 20:57:22.264284       9 sparkapp_metrics.go:144] Exporting metrics for test-envvars; old status: {  0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } { } map[] 0 0} new status: { 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}
I0225 20:57:22.264331       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:22.264355       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:22.264421       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{ 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}]
I0225 20:57:22.269993       9 controller.go:218] SparkApplication spark/test-envvars was updated, enqueueing it
I0225 20:57:22.270884       9 sparkapp_metrics.go:144] Exporting metrics for test-envvars; old status: { 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1} new status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}
I0225 20:57:22.270930       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:22.270952       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:22.271030       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}]
I0225 20:57:22.271160       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:22.659313       9 spark_pod_eventhandler.go:58] Pod test-envvars-driver updated in namespace spark.
I0225 20:57:22.659341       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-envvars for app update processing.
I0225 20:57:22.659366       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:22.659428       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}]
I0225 20:57:22.659580       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:23.886659       9 spark_pod_eventhandler.go:58] Pod test-envvars-driver updated in namespace spark.
I0225 20:57:23.886709       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-envvars for app update processing.
I0225 20:57:23.886728       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:23.886827       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {RUNNING } map[] 1 1}]
I0225 20:57:23.886958       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envvars", UID:"431a855d-e22b-4fee-a600-ac549e498fdc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965479", FieldPath:""}): type: 'Normal' reason: 'SparkDriverRunning' Driver test-envvars-driver is running
I0225 20:57:23.897144       9 controller.go:218] SparkApplication spark/test-envvars was updated, enqueueing it
I0225 20:57:23.902009       9 sparkapp_metrics.go:144] Exporting metrics for test-envvars; old status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUBMITTED } map[] 1 1} new status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {RUNNING } map[] 1 1}
I0225 20:57:23.902055       9 metrics.go:111] Incrementing spark_app_running_count with labels map[app_type:Unknown]
I0225 20:57:23.902066       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:23.902084       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:23.902141       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {RUNNING } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {RUNNING } map[] 1 1}]
I0225 20:57:23.902235       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:25.907916       9 spark_pod_eventhandler.go:58] Pod test-envvars-driver updated in namespace spark.
I0225 20:57:25.907941       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark/test-envvars for app update processing.
I0225 20:57:25.907986       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:25.908085       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {RUNNING } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25.908017314 +0000 UTC m=+43990.324860500 {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUCCEEDING } map[] 1 1}]
I0225 20:57:25.908217       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envvars", UID:"431a855d-e22b-4fee-a600-ac549e498fdc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965489", FieldPath:""}): type: 'Normal' reason: 'SparkDriverCompleted' Driver test-envvars-driver completed
I0225 20:57:25.943834       9 controller.go:218] SparkApplication spark/test-envvars was updated, enqueueing it
I0225 20:57:25.952313       9 sparkapp_metrics.go:144] Exporting metrics for test-envvars; old status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {RUNNING } map[] 1 1} new status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUCCEEDING } map[] 1 1}
I0225 20:57:25.952373       9 metrics.go:124] Decrementing spark_app_running_count with labels map[app_type:Unknown] metricVal to 0
I0225 20:57:25.952390       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:25.952402       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:25.952435       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUCCEEDING } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {COMPLETED } map[] 1 1}]
I0225 20:57:25.952617       9 event.go:221] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark", Name:"test-envvars", UID:"431a855d-e22b-4fee-a600-ac549e498fdc", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8965500", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationCompleted' SparkApplication test-envvars completed
I0225 20:57:25.980096       9 controller.go:218] SparkApplication spark/test-envvars was updated, enqueueing it
I0225 20:57:25.983001       9 sparkapp_metrics.go:144] Exporting metrics for test-envvars; old status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {SUCCEEDING } map[] 1 1} new status: {spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {COMPLETED } map[] 1 1}
I0225 20:57:25.983039       9 controller.go:265] Ending processing key: "spark/test-envvars"
I0225 20:57:25.983059       9 controller.go:258] Starting processing key: "spark/test-envvars"
I0225 20:57:25.983077       9 controller.go:591] Trying to update SparkApplication spark/test-envvars, from: [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {COMPLETED } map[] 1 1}] to [{spark-6367044d80f746a4a15af9a29f9fe767 2fc49748-4ecb-4935-aa4f-4cb66d8fc276 2020-02-25 20:57:22 +0000 UTC 2020-02-25 20:57:25 +0000 UTC {test-envvars-ui-svc 4040 10.20.9.31:4040   test-envvars-driver} {COMPLETED } map[] 1 1}]
I0225 20:57:25.983201       9 controller.go:265] Ending processing key: "spark/test-envvars"
picture rjonczy
👍2 👀1

Most helpful comment

You will need to enable the mutating webhook for env and envFrom to work.

picture liyinan926 on 25 Feb 2020
👍2

All 2 comments

You will need to enable the mutating webhook for env and envFrom to work.

liyinan926 picture liyinan926 on 25 Feb 2020
👍2

@liyinan926 many many thanks! It worked

As I am on private GKE, I've added:

enableWebhook: true
webhookPort: 443

and upgraded helm release and voila!

Env variable START_TIME exists. (START_TIME=1583121125371)
Env variable END_TIME exists. (END_TIME=1583124125371)
Env variable STEP_SIZE exists. (STEP_SIZE=60000)
Env variable WINDOW_SIZE exists. (WINDOW_SIZE=60000)
Env variable BATCH_SIZE exists. (BATCH_SIZE=10)
Env variable HOME exists. (HOME=/root)
rjonczy picture rjonczy on 26 Feb 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jemega picture jemega  Â·  8Comments
jdelgadillo picture jdelgadillo  Â·  8Comments
liyinan926 picture liyinan926  Â·  3Comments
liyinan926 picture liyinan926  Â·  7Comments
jgardner04 picture jgardner04  Â·  5Comments