Spark-on-k8s-operator: Security context not applied to driver/executor pods

Created on 4 Sep 2020  路  20Comments  路  Source: GoogleCloudPlatform/spark-on-k8s-operator

Trying to specify allowPrivilegeEscalation: false on driver/executor pods and securityContext isn't even getting set. Using spark-operator v1beta2-1.2.0-3.0.0.

kubectl apply -f pi.json

pi.json:
{ "apiVersion": "sparkoperator.k8s.io/v1beta2", "kind": "SparkApplication", "metadata": { "name": "pi", "namespace": "spark-operator" }, "spec": { "driver": { "cores": 1, "labels": { "version": "3.0.0" }, "memory": "472m", "securityContext": { "allowPrivilegeEscalation": false }, "serviceAccount": "spark-operator-spark" }, "executor": { "cores": 1, "instances": 1, "labels": { "version": "3.0.0" }, "memory": "472m", "securityContext": { "allowPrivilegeEscalation": false } }, "image": "gcr.io/spark-operator/spark:v3.0.0", "imagePullPolicy": "Always", "mainApplicationFile": "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar", "mainClass": "org.apache.spark.examples.SparkPi", "restartPolicy": { "type": "Never" }, "sparkVersion": "3.0.0", "type": "Scala" } }

allowPrivilegeEscalation is set on sparkapplication resource:
kubectl get sparkapplications -n spark-operator pi -o json
{ "apiVersion": "sparkoperator.k8s.io/v1beta2", "kind": "SparkApplication", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"sparkoperator.k8s.io/v1beta2\",\"kind\":\"SparkApplication\",\"metadata\":{\"annotations\":{},\"name\":\"pi\",\"namespace\":\"spark-operator\"},\"spec\":{\"driver\":{\"cores\":1,\"labels\":{\"version\":\"3.0.0\"},\"memory\":\"472m\",\"securityContext\":{\"allowPrivilegeEscalation\":false},\"serviceAccount\":\"spark-operator-spark\"},\"executor\":{\"cores\":1,\"instances\":1,\"labels\":{\"version\":\"3.0.0\"},\"memory\":\"472m\",\"securityContext\":{\"allowPrivilegeEscalation\":false}},\"image\":\"gcr.io/spark-operator/spark:v3.0.0\",\"imagePullPolicy\":\"Always\",\"mainApplicationFile\":\"local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar\",\"mainClass\":\"org.apache.spark.examples.SparkPi\",\"restartPolicy\":{\"type\":\"Never\"},\"sparkVersion\":\"3.0.0\",\"type\":\"Scala\"}}\n" }, "creationTimestamp": "2020-09-04T16:06:38Z", "generation": 1, "name": "pi", "namespace": "spark-operator", "resourceVersion": "42501604", "selfLink": "/apis/sparkoperator.k8s.io/v1beta2/namespaces/spark-operator/sparkapplications/pi", "uid": "5ee164fe-569a-4688-8608-45abd0ea5d7a" }, "spec": { "driver": { "cores": 1, "labels": { "version": "3.0.0" }, "memory": "472m", "securityContext": { "allowPrivilegeEscalation": false }, "serviceAccount": "spark-operator-spark" }, "executor": { "cores": 1, "instances": 1, "labels": { "version": "3.0.0" }, "memory": "472m", "securityContext": { "allowPrivilegeEscalation": false } }, "image": "gcr.io/spark-operator/spark:v3.0.0", "imagePullPolicy": "Always", "mainApplicationFile": "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar", "mainClass": "org.apache.spark.examples.SparkPi", "restartPolicy": { "type": "Never" }, "sparkVersion": "3.0.0", "type": "Scala" }, "status": { "applicationState": { "state": "COMPLETED" }, "driverInfo": { "podName": "pi-driver", "webUIAddress": "10....:4040", "webUIPort": 4040, "webUIServiceName": "pi-ui-svc" }, "executionAttempts": 1, "executorState": { "spark-pi-0b09457459deefb2-exec-1": "COMPLETED" }, "lastSubmissionAttemptTime": "2020-09-04T16:06:43Z", "sparkApplicationId": "spark-3ffa3d1e87a448bc8dc3f1959dd5fc99", "submissionAttempts": 1, "submissionID": "c59db62e-18f4-4952-8107-3273205a0d5b", "terminationTime": "2020-09-04T16:07:10Z" } }

But not set on driver (securityContext is completly missing):
kubectl get pods -n spark-operator pi-driver -o json
{ "apiVersion": "v1", "kind": "Pod", "metadata": { "annotations": { "cni.projectcalico.org/podIP": "10..../32" }, "creationTimestamp": "2020-09-04T16:06:42Z", "labels": { "spark-app-selector": "spark-3ffa3d1e87a448bc8dc3f1959dd5fc99", "spark-role": "driver", "sparkoperator.k8s.io/app-name": "pi", "sparkoperator.k8s.io/launched-by-spark-operator": "true", "sparkoperator.k8s.io/submission-id": "c59db62e-18f4-4952-8107-3273205a0d5b", "version": "3.0.0" }, "name": "pi-driver", "namespace": "spark-operator", "ownerReferences": [ { "apiVersion": "sparkoperator.k8s.io/v1beta2", "controller": true, "kind": "SparkApplication", "name": "pi", "uid": "5ee164fe-569a-4688-8608-45abd0ea5d7a" } ], "resourceVersion": "42501598", "selfLink": "/api/v1/namespaces/spark-operator/pods/pi-driver", "uid": "38a4fc2d-db61-41cc-9440-7d79284292d8" }, "spec": { "containers": [ { "args": [ "driver", "--properties-file", "/opt/spark/conf/spark.properties", "--class", "org.apache.spark.examples.SparkPi", "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar" ], "env": [ { "name": "SPARK_USER", "value": "root" }, { "name": "SPARK_APPLICATION_ID", "value": "spark-3ffa3d1e87a448bc8dc3f1959dd5fc99" }, { "name": "SPARK_DRIVER_BIND_ADDRESS", "valueFrom": { "fieldRef": { "apiVersion": "v1", "fieldPath": "status.podIP" } } }, { "name": "SPARK_LOCAL_DIRS", "value": "/var/data/spark-132355e9-1754-4f49-8a7f-79935c9dda1c" }, { "name": "SPARK_CONF_DIR", "value": "/opt/spark/conf" } ], "image": "gcr.io/spark-operator/spark:v3.0.0", "imagePullPolicy": "Always", "name": "spark-kubernetes-driver", "ports": [ { "containerPort": 7078, "name": "driver-rpc-port", "protocol": "TCP" }, { "containerPort": 7079, "name": "blockmanager", "protocol": "TCP" }, { "containerPort": 4040, "name": "spark-ui", "protocol": "TCP" } ], "resources": { "limits": { "memory": "856Mi" }, "requests": { "cpu": "1", "memory": "856Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/var/data/spark-132355e9-1754-4f49-8a7f-79935c9dda1c", "name": "spark-local-dir-1" }, { "mountPath": "/opt/spark/conf", "name": "spark-conf-volume" }, { "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount", "name": "spark-operator-spark-token-4r6lx", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "enableServiceLinks": true, "nodeName": "k8s-merlin-w02", "priority": 0, "restartPolicy": "Never", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "spark-operator-spark", "serviceAccountName": "spark-operator-spark", "terminationGracePeriodSeconds": 30, "tolerations": [ { "effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists", "tolerationSeconds": 300 }, { "effect": "NoExecute", "key": "node.kubernetes.io/unreachable", "operator": "Exists", "tolerationSeconds": 300 } ], "volumes": [ { "emptyDir": {}, "name": "spark-local-dir-1" }, { "configMap": { "defaultMode": 420, "name": "pi-fa08b17459deb244-driver-conf-map" }, "name": "spark-conf-volume" }, { "name": "spark-operator-spark-token-4r6lx", "secret": { "defaultMode": 420, "secretName": "spark-operator-spark-token-4r6lx" } } ] }, "status": { "conditions": [ { "lastProbeTime": null, "lastTransitionTime": "2020-09-04T16:06:42Z", "reason": "PodCompleted", "status": "True", "type": "Initialized" }, { "lastProbeTime": null, "lastTransitionTime": "2020-09-04T16:07:10Z", "reason": "PodCompleted", "status": "False", "type": "Ready" }, { "lastProbeTime": null, "lastTransitionTime": "2020-09-04T16:07:10Z", "reason": "PodCompleted", "status": "False", "type": "ContainersReady" }, { "lastProbeTime": null, "lastTransitionTime": "2020-09-04T16:06:42Z", "status": "True", "type": "PodScheduled" } ], "containerStatuses": [ { "containerID": "docker://f79dc01dc339b102eee510ebf9bd1757ba2f5c717cea2b56be28b2a308bfab33", "image": "gcr.io/spark-operator/spark:v3.0.0", "imageID": "docker-pullable://gcr.io/spark-operator/spark@sha256:cbb1e870da3b1555936a12ddc1a57889c63e02c42c611c3dba349e81d9fd329e", "lastState": {}, "name": "spark-kubernetes-driver", "ready": false, "restartCount": 0, "state": { "terminated": { "containerID": "docker://f79dc01dc339b102eee510ebf9bd1757ba2f5c717cea2b56be28b2a308bfab33", "exitCode": 0, "finishedAt": "2020-09-04T16:07:08Z", "reason": "Completed", "startedAt": "2020-09-04T16:06:52Z" } } } ], "hostIP": "10...", "phase": "Succeeded", "podIP": "10....", "qosClass": "Burstable", "startTime": "2020-09-04T16:06:42Z" } }

Any idea?

Thanks

picture doctapp

Most helpful comment

@jinxingwang appreciate if you can make a fix. Thanks!

picture liyinan926 on 17 Sep 2020
👍2

All 20 comments

Do you have the mutating admission webhook deployed in your cluster?
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/de0bdf1275e77b6d053d408c045c427aa94a42ab/docs/user-guide.md#using-pod-security-context
If not, securityContextwill not be changed. If so, look for the webhook log, and see how it is processing the securityContext.

jinxingwang picture jinxingwang on 16 Sep 2020

Yes, we use the webhook to mount volumes and other things.

doctapp picture doctapp on 16 Sep 2020

@jinxingwang I've checked the logs and there's no reference to securityContext. Can you try it out? What should I be expecting? Thanks

doctapp picture doctapp on 16 Sep 2020

Can you post webhook logs about your sparkapplication?

jinxingwang picture jinxingwang on 16 Sep 2020

Sorry, can't since its intermingled with private stuff.

doctapp picture doctapp on 16 Sep 2020

The log from webhook is the key to debug this. Please find a way to check the log of your webhook. What it is doing is, it will create a patch to modify the pod/executor spec, you should check if there such a patch exists, and what it is patching? Source code here:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/9d4592a0fc69a70fced060be17ce589620c26c67/pkg/webhook/patch.go#L529-L541

jinxingwang picture jinxingwang on 16 Sep 2020

Doesn't seem to be called since I can't see a patch from the logs and there's no ref to the securityContext. Here are some logs until it gets running:

I0916 18:45:53.835471       9 controller.go:179] SparkApplication spark-operator/pi was added, enqueueing it for submission
I0916 18:45:53.835642       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:45:53.835902       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"0b6982fb-0f53-4291-9816-9df6e99eef6a", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"44928567", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationAdded' SparkApplication pi was added, enqueuing it for submission
I0916 18:45:53.835939       9 submission.go:65] spark-submit arguments: [/opt/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master k8s://https://10....:443 --deploy-mode cluster --conf spark.kubernetes.namespace=spark-operator --conf spark.app.name=pi --conf spark.kubernetes.driver.pod.name=pi-driver --conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v3.0.0 --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=pi --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=1297f1e3-7a8f-4988-a8c2-c7f34cb0495b --conf spark.driver.cores=1 --conf spark.driver.memory=472m --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator-spark --conf spark.kubernetes.driver.label.version=3.0.0 --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=pi --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=1297f1e3-7a8f-4988-a8c2-c7f34cb0495b --conf spark.executor.instances=1 --conf spark.executor.cores=1 --conf spark.executor.memory=472m --conf spark.kubernetes.executor.label.version=3.0.0 local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar]
I0916 18:45:58.899029       9 webhook.go:246] Serving admission request
I0916 18:45:58.900307       9 webhook.go:556] Pod pi-driver in namespace spark-operator is subject to mutation
I0916 18:45:58.919243       9 spark_pod_eventhandler.go:47] Pod pi-driver added in namespace spark-operator.
I0916 18:45:58.929447       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0916 18:45:58.957349       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0916 18:45:59.920055       9 controller.go:671] SparkApplication spark-operator/pi has been submitted
I0916 18:45:59.920132       9 sparkui.go:151] Creating a service pi-ui-svc for the Spark UI for application pi
I0916 18:45:59.920295       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"0b6982fb-0f53-4291-9816-9df6e99eef6a", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"44928567", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSubmitted' SparkApplication pi was submitted successfully
I0916 18:45:59.953327       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "lastSubmissionAttemptTime": null,
  "terminationTime": null,
  "driverInfo": {},
  "applicationState": {
    "state": ""
  }
}
to:
{
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10...:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0916 18:45:59.974102       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0916 18:45:59.976661       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:45:59.976699       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:45:59.976870       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10...:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10...:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0916 18:45:59.993053       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:45:59.993421       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0916 18:45:59.993457       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:45:59.993604       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:01.729992       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0916 18:46:01.730044       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:01.730104       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:01.730337       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:03.869501       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0916 18:46:03.869538       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:03.869577       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:03.869775       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"0b6982fb-0f53-4291-9816-9df6e99eef6a", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"44928596", FieldPath:""}): type: 'Normal' reason: 'SparkDriverRunning' Driver pi-driver is running
I0916 18:46:03.869881       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10....:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10....:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0916 18:46:03.891304       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0916 18:46:03.893637       9 metrics.go:112] Incrementing spark_app_running_count with labels map[app_type:Unknown]
I0916 18:46:03.893695       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:03.893723       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:03.893872       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:09.390998       9 webhook.go:246] Serving admission request
I0916 18:46:09.392052       9 webhook.go:556] Pod spark-pi-3f6f1474983cfaae-exec-1 in namespace spark-operator is subject to mutation
I0916 18:46:09.404363       9 spark_pod_eventhandler.go:47] Pod spark-pi-3f6f1474983cfaae-exec-1 added in namespace spark-operator.
I0916 18:46:09.404397       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:09.404439       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:09.404873       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10....:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10...:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-3f6f1474983cfaae-exec-1": "PENDING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0916 18:46:09.404962       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"0b6982fb-0f53-4291-9816-9df6e99eef6a", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"44928619", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-3f6f1474983cfaae-exec-1 is pending
I0916 18:46:09.421550       9 spark_pod_eventhandler.go:58] Pod spark-pi-3f6f1474983cfaae-exec-1 updated in namespace spark-operator.
I0916 18:46:09.421607       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:09.426934       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0916 18:46:09.428238       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:09.428293       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:09.428475       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:09.457695       9 spark_pod_eventhandler.go:58] Pod spark-pi-3f6f1474983cfaae-exec-1 updated in namespace spark-operator.
I0916 18:46:09.457760       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:09.457815       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:09.458126       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:11.015847       9 spark_pod_eventhandler.go:58] Pod spark-pi-3f6f1474983cfaae-exec-1 updated in namespace spark-operator.
I0916 18:46:11.015882       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:11.015927       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:11.016120       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0916 18:46:13.000132       9 spark_pod_eventhandler.go:58] Pod spark-pi-3f6f1474983cfaae-exec-1 updated in namespace spark-operator.
I0916 18:46:13.000164       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0916 18:46:13.000205       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0916 18:46:13.000484       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10...:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-3f6f1474983cfaae-exec-1": "PENDING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-86b1815384be4830a0b989d5f397399d",
  "submissionID": "1297f1e3-7a8f-4988-a8c2-c7f34cb0495b",
  "lastSubmissionAttemptTime": "2020-09-16T18:45:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10....:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-3f6f1474983cfaae-exec-1": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
doctapp picture doctapp on 16 Sep 2020

This looks like spark-operator log. you need logs from the webhook pod.

jinxingwang picture jinxingwang on 16 Sep 2020

Weird since I don't see a running webhook pod, only a completed init pod, a service, and a job.batch:

service/spark-operator-webhook ClusterIP 10.43.157.219 <none> 443/TCP 12d
spark-operator-sparkoperator-webhook-init-dj4wn 0/1 Completed 0 12d
job.batch/spark-operator-sparkoperator-webhook-init 1/1 22s 12d

We're installing using --set enableWebhook=true and --set webhookPort=443.

How can volumes get mounted then? How can I debug the startup of the webhook?

Thanks

doctapp picture doctapp on 16 Sep 2020

This is where you don't have a webhook deploy. you can try research on how to use webhook.

jinxingwang picture jinxingwang on 16 Sep 2020

Sorry not following. We deploy with the web hook enabled and we use features that are only enabled through the web hook and they work (like mounting secrets/configmaps on driver and executor pods).

So, how come the webhook pod isn't running while these features work?

How can I debug this process (helm didn't complain about anything, no error messages in webhook init...)?

doctapp picture doctapp on 16 Sep 2020

is spark-operator-sparkoperator-webhook-init the long running webhook pod?
why you have a webhook that is complete? spark-operator-sparkoperator-webhook-init-dj4wn
You should be able to log on webhook pod, and see logs about your job. can you try that?

jinxingwang picture jinxingwang on 16 Sep 2020

Could you please point me to the yaml file that is starting the webhook pod you mentioned?

I'm looking at https://github.com/helm/charts/blob/master/incubator/sparkoperator/templates/spark-operator-deployment.yaml which looks like the sparkoperator container which is running the webhook and I provided its logs above. webhook-init is just a job generating the cert.

I just did a sample deployment again and ran the pi.json above and same problem occurs, no securityContext set.

Can you please try reproducing it?

Thanks

doctapp picture doctapp on 17 Sep 2020

Here are the full logs on a brand new cluster:

$ kubectl logs -f -n spark-operator spark-operator-sparkoperator-webhook-init-vl8gj
Generating certs for the Spark pod admission webhook in /tmp/spark-pod-webhook-certs.
Generating RSA private key, 2048 bit long modulus (2 primes)
............................+++++
....................................................+++++
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus (2 primes)
........................................+++++
......+++++
e is 65537 (0x010001)
Signature ok
subject=CN = spark-operator-webhook.spark-operator.svc
Getting CA Private Key
Creating a secret for the certificate and keys
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15861    0  8040  100  7821   413k   401k --:--:-- --:--:-- --:--:--  815k
HTTP/2 201 
audit-id: 888a25f7-e973-45e7-b94c-2780ac068fdd
content-type: application/json
date: Thu, 17 Sep 2020 12:54:53 GMT

{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "spark-webhook-certs",
    "namespace": "spark-operator",
    "selfLink": "/api/v1/namespaces/spark-operator/secrets/spark-webhook-certs",
    "uid": "7c8e3411-8975-48f1-a099-d0275179f7f9",
    "resourceVersion": "1783",
    "creationTimestamp": "2020-09-17T12:54:53Z"
  },
  "data": {
    "ca-cert.pem": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLekNDQWhPZ0F3SUJBZ0lVQ1JwYUdDcmFMWUo5T3NwTmhYQkNDUVJuTTZVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pERWlNQ0FHQTFVRUF3d1pjM0JoY21zdGIzQmxjbUYwYjNJdGQyVmlhRzl2YTE5allUQWdGdzB5TURBNQpNVGN4TWpVME5USmFHQTh5TWprME1EY3dNekV5TlRRMU1sb3dKREVpTUNBR0ExVUVBd3daYzNCaGNtc3RiM0JsCmNtRjBiM0l0ZDJWaWFHOXZhMTlqWVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUIKQU1BdXk1N3BiWk80elVHQTBXMU1PNmhJZVdLamliQkZyQVpBNk9ZMG5IRzJjVGI3QnY2VUhKNTRGdnhwZmlSWgovWXd5MVlGbkNPdGNBK2NWNjhQRmxtRWxRT2oydzZScmVPaHJDSm90bXNLZWxNemRzd2tyank0bU82bW5OdWJuCkpSL2k3Znh2akdkR2poclR5aFhUMVpSNkcxbTk3MjVjcWNwL1dLblZiQjJ5S0lGUjg4TUNCSGdtdElHejRIWksKak41NjlTS01NaHdsOUJVT2kxTXJ5bWRVaXJZYWx0MVJ5a3NmWTQ3cXNnTHlBbmFvL2RZZWlMQlhNV29IQkpucQo1aHVQcUVuSUFYMkpEUmtwcGQyVG9FL1JFT1E3eVFDcHJCN3E4ZVhWbFcwS2hiZFVKRnBQQlpIcS9rK29aQ2VqCmpMeFJ6TXhKQTNmczhrbGI1YzJTQmQwQ0F3RUFBYU5UTUZFd0hRWURWUjBPQkJZRUZNY1h0bmljYWJZa3BDSmYKUU5WQVRveTNKN1ZoTUI4R0ExVWRJd1FZTUJhQUZNY1h0bmljYWJZa3BDSmZRTlZBVG95M0o3VmhNQThHQTFVZApFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUh5QjRtbUhaTkVVRnlzNm14eU9jSHQyCnV0SVBON1hVbFlaY1pTdjhwcjZMTHE4M2k1ZGNIODgyN1pXblVpWVFhblpBaUoxZVlYUXVJaFZKNTNDYVZqRC8KNmthS2RTSTd6RXZLZHRaZ21MYTRZb1RCVWJIcGgvU041NGw1ZzZCaFBsNXdBU3IzL2tVdDErai9YN01CV1NBMQp5M1RwVERGaERvSTNKZkllaU02ZWpkWm5RUVJpVlplT01LUXU1eENnMkdPZ2xua2x4QkZFSW4wcFIvSnFoZC9RCnlVaE5uMVA4bVNNd3BZa2pPSWg5Y0sxMm5UR3pzUDhtVjAzSXBLdzJRaHBpT2tybGZvK0V0bXBqTkszZGlMRFkKSXVndTd5ZlJuUDJhWEpHdWNBcUxkbzdPZDI1UXdYNHBlWnBLbVhqaStNNmhTa1BTSXZ2d1FndVFFNkJ5eXVzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
    "ca-key.pem": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBd0M3TG51bHRrN2pOUVlEUmJVdzdxRWg1WXFPSnNFV3NCa0RvNWpTY2NiWnhOdnNHCi9wUWNubmdXL0dsK0pGbjlqRExWZ1djSTYxd0Q1eFhydzhXV1lTVkE2UGJEcEd0NDZHc0ltaTJhd3A2VXpOMnoKQ1N1UExpWTdxYWMyNXVjbEgrTHQvRytNWjBhT0d0UEtGZFBWbEhvYldiM3ZibHlweW45WXFkVnNIYklvZ1ZIegp3d0lFZUNhMGdiUGdka3FNM25yMUlvd3lIQ1gwRlE2TFV5dktaMVNLdGhxVzNWSEtTeDlqanVxeUF2SUNkcWo5CjFoNklzRmN4YWdjRW1lcm1HNCtvU2NnQmZZa05HU21sM1pPZ1Q5RVE1RHZKQUttc0h1cng1ZFdWYlFxRnQxUWsKV2s4RmtlcitUNmhrSjZPTXZGSE16RWtEZCt6eVNWdmx6WklGM1FJREFRQUJBb0lCQVFDbU4xTHhMRjhUaHFBdgo2U1E0N2RnaklodXFUbHRGMDl5R3hCa3JFWkhGWDgvbmg1RnpHZEx1MUZxOUVOaUN4RHBneVk2VGp2UE43QldNCkJnK1ZrUTBSRkZGaHJCdGwxM1lKUHBtQUQ3VnNZZys4YUhyeWtYUVNyQXFxSHUrRElhT2dRbnNtKzZ0cG4wSkUKdmF4bzJJQS9IS1VQSnhFd1dJQ1puUzZvSk9pUGZRa3NpaUxsQkN2M0J0eUd0Y0wwMlZhVVVCUUFtNXVpM2tteQoyRnVCSlIzNDNBbWMraTIrY0VjcVJxRzZKOTdaM3pMTmJLMmxyQTcxMHp2eG1DbFJZZktzRCtibFE0UVpQSE1VCkJROE5pTEk0RW4xeDhtdXpyTURhZGlZREdPbXAwWG5BcDZhOUJkcHp2bzlGT2lzM2RUdW9aT2QrZzNrLzJQWU4KL0lPbHNrQ0JBb0dCQU9nTHRMS2hiMzkwTUhza0UxM05NMDZ5cC9rYUlIK0VZTVlGUlZkYUVyMEIzWERYZGoxMwpZWGExZ3laR3ZvdkFKTDFjaCtaOHJxd0xNZDhidnFoWDA1YjEwT1R0b1crakpIY2FXMWc1czR5MmpHSjhwRmpWClVlbUo5cENIWnE4RGV2TXM0czRHV2JRMHliRGhwWVcwczNnK1pWRXNWdUxQUllMZ0NQNWhiMk10QW9HQkFOUUYKb013LytOdDdSZU1Geko2S3l4U1lOTk93YUFkNDdyT05rOEVLa3RpaXRUb1RoVzkvc1BzZE92M2dkWWNjd1ZmeApHR295YzlldnhIMksxRkg5YUpoMVArMjFzUUtVWlJjZ2VIN1NRcGNGOEhZT1JuekN1emZwd0IrSUp0bkJCNkZGCjFQYVlReE14UjdKUFAzTkVVVGJhd3NkYjNHQ21qR1o3a1ZCSXA1dHhBb0dCQUxpM3FNNlNLRG5qZWp4c2lXWWcKVlhtT1NwekhxWnNZL0JobVZiNmxtMGtPVlFMZzRlVW9SVlBFUm5TUDJGdFRSMnZlWHVpaDFFSnNYdlRtRnkzWApURDVkekQ2UTAvUVVnSkMzVUlPNDJ5OW1tZlpFbG1lSTJMMzU3MmM2QWduTDBYSUhCejFPWE5aUk4wVXhPeHhtCnJuOUgzRXR6Nm01KzVTeTd2eTBRenlTNUFvR0FYeDZ2UnpNa0dJT3psT2hUSm9XcUduRTJzZ29nZ2NtMWZZUjAKMTFSQkVVUGQvbEludVNBV2JISnV0dHR1bnRFOXhKQjduWmxJODJqOXhEejJpRm9pQk5RRmZFWFpSZ2tCV1dETgo3a2V5YkVMVDRJTURDSkhzb0djVU5KRkdyTWEyLzVsdXM3c1h6Sko5VWZmUHhZek95c2RXUDA1dERrZHFFY2c2CkowU1JQV0VDZ1lBdVVTM2FTcjNBZDR3d3lsQlBjQnFhUFRqa01wbFQwRkVlM3kyMWt5K1J1aklzdlNYQVVzS28KK0c5Snd5SHdwZlJUb1JSUkpHVGNzNnRkV3I2eE9yb1luRVEzNHByU21aVWxvUXZWUUVZREdXMmdXdzRKNjV2cgpKQjFjQXMzNWtZSXUwQUsvMEJBUnE2Y0kzSmgyT2tIbGlnZkplb3JLdE1icnhVV3c0NVc0Nnc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=",
    "server-cert.pem": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lVTjRWZW1Zb2VJTlhWcTJpRW1OZml0NEttMUpjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pERWlNQ0FHQTFVRUF3d1pjM0JoY21zdGIzQmxjbUYwYjNJdGQyVmlhRzl2YTE5allUQWdGdzB5TURBNQpNVGN4TWpVME5UTmFHQTh5TWprME1EY3dNekV5TlRRMU0xb3dOREV5TURBR0ExVUVBd3dwYzNCaGNtc3RiM0JsCmNtRjBiM0l0ZDJWaWFHOXZheTV6Y0dGeWF5MXZjR1Z5WVhSdmNpNXpkbU13Z2dFaU1BMEdDU3FHU0liM0RRRUIKQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURjeEN2QnNmVG1Xa0hzbnFSL2cxNkppSkVVZFBlZkorUnhlMTFLcXVldApVVm1pWXJlWVZSdzdTWkMwbW5IZml5NXVUZlU1WXZmYTA1aEVEZ1laUkFacEptWUhvekZXdytDbE5rNWs4MlZTCkh5Q1VKMDZNYnplQURRQlpxeEpGNjRWVEVvQlJOMVlmUm9WTXFDR014REZXK2hFd0tNSVNKR2VkV3ZFbVJ6WTMKVWtvVWVCVHR3cUtwQlFuM1FFMlpaNUpXeHpDZVJsN1l6bkQ3bzRsTld0V2lPU3VaRWtjUmYrYUQrSmJxQUwyagorZnRLUms0Ni95YVBad2ozQ0cvWkx5OTcvSG41VDNFRFdJcEl3RnN2ZkhZN2ozMmttRkpVVkdQamZrT2lkTEJRCnJCRWgvamVncnBRbm5WNHF0b0dML24yR1MwaGR0N1UyK05mMFlmYlYwWXN4QWdNQkFBR2pPVEEzTUFrR0ExVWQKRXdRQ01BQXdDd1lEVlIwUEJBUURBZ1hnTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQ0JnZ3JCZ0VGQlFjRApBVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBR010QmZDa1hRSHFmVUx0R2hCU0FWSUd4VlBjdHJqdUt6cHp6CjF2OUF3Mnd1WTJFS2NYVzFpOWNoZTJDQlZ6c25nN0RtNFRLTEVONjh6c3duRFRkM01BUlo1Nm8zMklueUJ2NnYKSnJoc0IvYUkxc014Q2YzbTloSFpsOHYrbDRuY2EwOTJRTTFLOHgrMDNITmVBU1ZlZllnRmRVMlV6V0pWSlhrNQpzWWZXSitQK1FwNDhVVTllZkUrZlJKRGxUNXcwenNlMXpXd3RQcVlPSDVCOWpUOERYNnNWZ0N1dUpvcEMrL3NHCll3Q3ZsT25oZEgwMzVaTVB0L3V6dHRjSld1MXBYc0wrL1kxamRQbkVIaFpFM0srOHJNQTcvb3VMRmZTTDRJR0EKUnRkRlJvNzRsVEtockdrbjFvWGRpakZGVWljeHkrY0tKOWVFKzNVTm51Ry9xWDFFNXc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
    "server-key.pem": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBM01RcndiSDA1bHBCN0o2a2Y0TmVpWWlSRkhUM255ZmtjWHRkU3FybnJWRlpvbUszCm1GVWNPMG1RdEpweDM0c3ViazMxT1dMMzJ0T1lSQTRHR1VRR2FTWm1CNk14VnNQZ3BUWk9aUE5sVWg4Z2xDZE8Kakc4M2dBMEFXYXNTUmV1RlV4S0FVVGRXSDBhRlRLZ2hqTVF4VnZvUk1DakNFaVJublZyeEprYzJOMUpLRkhnVQo3Y0tpcVFVSjkwQk5tV2VTVnNjd25rWmUyTTV3KzZPSlRWclZvamtybVJKSEVYL21nL2lXNmdDOW8vbjdTa1pPCk92OG1qMmNJOXdodjJTOHZlL3g1K1U5eEExaUtTTUJiTDN4Mk80OTlwSmhTVkZSajQzNURvblN3VUt3UklmNDMKb0s2VUo1MWVLcmFCaS81OWhrdElYYmUxTnZqWDlHSDIxZEdMTVFJREFRQUJBb0lCQVFDTGVQUWIvdzIxcWpnVQpBWFh6aVU5a25IN3I2YW0zUjBDdjRUeGpPKzM2TlhibmZHNTVNdXdQR0w3dkd0UHVZc1FHdEFIRks1MWpCbU1nCk82dm1zbTlLWnhQMnAwYzRoYStiTU4vK0x6ZEgxcHFDZndMRS9rNlYrNkRKQmFjWnZWNFJ5WkovdlBJUCtFR20KMkdHNXlRbFM1ZEdPRjNIa2ZGTnB1bGEzMUMyUWJLUUZMM0pEWi9MZlozMnFHT0VZSXlnWmZFZzNNRUlBcWQ3RwpFdFFjdzhFUENmeHdvTHkxcGdGODFBN2REOE1JZ2JiM0grRDkzVkNuUU5jNXNKaCtvODE1RGsxK0lZKzNsMUdUCmptT3FKZGl6dkhsc0Y1UkNzWFc3NEozR0pVQyswN1pGY1R4c1FRaHdyQTIyeDNKazZXQ3QwUjB4aDBjZGtpbDIKNVoyNUhOMHhBb0dCQVBaYWFkNG1qYkpOV2ZISEljUHZ4TnozaVpNSitNWTJDOHBTelNsWGtoYlpPeUdweWJDQwpTcSt3TE4vRkIycW5Vc1NKVnZwOXdnVFQyYkRVbWQ4UUtXS0lSWUxPS2plUlg2VmRVZzFUQy8yVUV6NG0yRjJxCmtwdEk0QmNDcDF2RjRZQjM2MlBSakFyRkJxazRTanhlNzRjWHFVK1MxQTkxdW1rSGFBY2cwRWFEQW9HQkFPVnAKUW5uVlhVL2djSGttNXo0QVhtd1dZTXhzd1B6ckFNZnk4dk9RQ2JSbnVrYk8rZlVGUXQ4Y1JvTGhCT1hxNUJ0cgpjWjAwK2ZnVW9KSUVLVURVNnJmc3NUWkJ4K3lZbGEvemEyUCtLaEVSTlB6N3pWdTArWUNBajRYWEFueG8xTjhiCldVYnRUSUNlbGRPU2dXSk8xL2xJQTkzcTF3ZTh5TVBNanZlT0Q1azdBb0dBVlJUQXYzSjlPWGt4ZVNrVzVqNm0KdE5IZzk2RllBOHlpb2N1YkE0NDA3Q00ySXRncXlnVVZtanUwSTh3K1ZYMVlQbnhZUjhhcS9WUFZPUmNUMVZBVgptdXdKWmJXaUt2TFFINkp0TmlEYVlVZUVwZEJ3cFZqRnJVK3I1UWFMbWt4ckp4MURkSVBvWThmTUZ5RlRzaW52CmU2NldVTjhhMGdRTXp6MDVsa3lrNGRVQ2dZQWc1Ti9nUWtTMzhGR1FQWkRJcGhyajJxMlVnaitGeEZoL1V3bWMKL05rM1lld212UVRBS2puMWJjTjZmck5LR0UwTzFYQWIvMzJjSWJFSVhzcGZmbWdOYVVYYnBzRDQzc1FQejVKbAphd3N6R1VtRXdNSW9WU0cxV0RtOEVpRk1IMmZ1ZWFocWNCb1pFeVFlaXhJOE8xdmlaWDRRY3R6VVFqQnpTc0J4CkpKQitBd0tCZ1FDRjU1RjdObXRMR09TemhzNHhtQTJiNzhlMmEvL3V6MjlSTVVOZGJiazQvQXordGhRaUVjWWQKV1FqcGRPdDNBQWl0eWRvTGx4bCtqYytuQ3o5Z0VYR3pXSGM0VE50M0tDbG1zQi8za3N0SmJkb2tjQ2tsUnBDUQpMZlZOSzhoeGhvbk1Kc24wV1h2Qk14Mkx1SHVpVStuaWd0K2JtaHpaeGtLbXRheDdUbVFmYUE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo="
  },
  "type": "Opaque"
}
Success - secret created.

Deleting /tmp/spark-pod-webhook-certs.

-----------------------------------------------

$ kubectl logs -f -n spark-operator spark-operator-sparkoperator-669d6f6d86-8jqk8
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ echo 0
+ echo 0
+ echo root:x:0:0:root:/root:/bin/bash
0
0
root:x:0:0:root:/root:/bin/bash
+ [[ -z root:x:0:0:root:/root:/bin/bash ]]
+ exec /usr/bin/tini -s -- /usr/bin/spark-operator -v=2 -namespace=spark-operator -ingress-url-format= -controller-threads=10 -resync-interval=30 -logtostderr -enable-metrics=true -metrics-labels=app_type -metrics-port=10254 -metrics-endpoint=/metrics -metrics-prefix= -enable-webhook=true -webhook-svc-namespace=spark-operator -webhook-port=443 -webhook-svc-name=spark-operator-webhook -webhook-config-name=spark-operator-sparkoperator-webhook-config -webhook-namespace-selector=
I0917 12:55:08.291748       9 main.go:143] Starting the Spark Operator
I0917 12:55:08.292185       9 main.go:176] Enabling metrics collecting and exporting to Prometheus
I0917 12:55:08.292280       9 metrics.go:142] Started Metrics server at localhost:10254/metrics
I0917 12:55:08.294163       9 webhook.go:218] Starting the Spark admission webhook server
I0917 12:55:08.311517       9 webhook.go:412] Creating a MutatingWebhookConfiguration for the Spark pod admission webhook
I0917 12:55:08.323739       9 main.go:218] Starting application controller goroutines
I0917 12:55:08.323791       9 controller.go:160] Starting the workers of the SparkApplication controller
I0917 12:55:08.323830       9 controller.go:96] Starting the ScheduledSparkApplication controller
I0917 12:55:08.424216       9 controller.go:102] Starting the workers of the ScheduledSparkApplication controller
I0917 12:55:16.706627       9 webhook.go:246] Serving admission request
I0917 12:55:16.713278       9 webhook.go:540] Pod  in namespace spark-operator is not subject to mutation
I0917 12:55:16.806480       9 webhook.go:246] Serving admission request
I0917 12:55:16.807174       9 webhook.go:540] Pod  in namespace spark-operator is not subject to mutation
I0917 13:19:56.186166       9 controller.go:179] SparkApplication spark-operator/pi was added, enqueueing it for submission
I0917 13:19:56.186237       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:19:56.186466       9 submission.go:65] spark-submit arguments: [/opt/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master k8s://https://10.1.1.1:443 --deploy-mode cluster --conf spark.kubernetes.namespace=spark-operator --conf spark.app.name=pi --conf spark.kubernetes.driver.pod.name=pi-driver --conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v3.0.0 --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=pi --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=5c848289-cb61-4118-b725-6d7ed8461222 --conf spark.driver.cores=1 --conf spark.driver.memory=472m --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-operator-spark --conf spark.kubernetes.driver.label.version=3.0.0 --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=pi --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=5c848289-cb61-4118-b725-6d7ed8461222 --conf spark.executor.instances=1 --conf spark.executor.cores=1 --conf spark.executor.memory=472m --conf spark.kubernetes.executor.label.version=3.0.0 local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar]
I0917 13:19:56.186484       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"7619", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationAdded' SparkApplication pi was added, enqueuing it for submission
I0917 13:19:59.430665       9 webhook.go:246] Serving admission request
I0917 13:19:59.431396       9 webhook.go:556] Pod pi-driver in namespace spark-operator is subject to mutation
I0917 13:19:59.453397       9 spark_pod_eventhandler.go:47] Pod pi-driver added in namespace spark-operator.
I0917 13:19:59.457177       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0917 13:19:59.959195       9 controller.go:671] SparkApplication spark-operator/pi has been submitted
I0917 13:19:59.959277       9 sparkui.go:151] Creating a service pi-ui-svc for the Spark UI for application pi
I0917 13:19:59.959437       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"7619", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationSubmitted' SparkApplication pi was submitted successfully
I0917 13:19:59.972425       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "lastSubmissionAttemptTime": null,
  "terminationTime": null,
  "driverInfo": {},
  "applicationState": {
    "state": ""
  }
}
to:
{
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:19:59.980563       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:19:59.981144       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:19:59.981196       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:19:59.981380       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:19:59.986387       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:19:59.987575       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:19:59.987612       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:19:59.987713       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:20:34.136894       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0917 13:20:34.136935       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:20:34.137009       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:20:34.137216       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:20:34.175353       9 webhook.go:246] Serving admission request
I0917 13:20:34.176168       9 webhook.go:540] Pod  in namespace kube-system is not subject to mutation
I0917 13:20:34.177283       9 webhook.go:246] Serving admission request
I0917 13:20:34.177999       9 webhook.go:540] Pod  in namespace kube-system is not subject to mutation
I0917 13:20:34.253226       9 webhook.go:246] Serving admission request
I0917 13:20:34.254623       9 webhook.go:540] Pod kube-proxy-gke-dst-gke-spark-1-dst-gke-spark-1-s-7c60373c-gqvf in namespace kube-system is not subject to mutation
I0917 13:20:34.698391       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0917 13:20:34.698427       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:20:34.698482       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:20:34.698645       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:20:40.934732       9 webhook.go:246] Serving admission request
I0917 13:20:40.935995       9 webhook.go:540] Pod  in namespace kube-system is not subject to mutation
I0917 13:21:05.625285       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0917 13:21:05.625319       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:05.625347       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:05.625645       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUBMITTED"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:21:05.625637       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"7644", FieldPath:""}): type: 'Normal' reason: 'SparkDriverRunning' Driver pi-driver is running
I0917 13:21:05.636674       9 metrics.go:112] Incrementing spark_app_running_count with labels map[app_type:Unknown]
I0917 13:21:05.636820       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:05.642120       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:21:05.642160       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:05.642292       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:11.070568       9 webhook.go:246] Serving admission request
I0917 13:21:11.071244       9 webhook.go:556] Pod spark-pi-c5607e749c39d210-exec-1 in namespace spark-operator is subject to mutation
I0917 13:21:11.085401       9 spark_pod_eventhandler.go:47] Pod spark-pi-c5607e749c39d210-exec-1 added in namespace spark-operator.
I0917 13:21:11.085436       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:11.085481       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:11.085734       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "PENDING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:21:11.085943       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"7935", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorPending' Executor spark-pi-c5607e749c39d210-exec-1 is pending
I0917 13:21:11.096240       9 spark_pod_eventhandler.go:58] Pod spark-pi-c5607e749c39d210-exec-1 updated in namespace spark-operator.
I0917 13:21:11.096284       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:11.100323       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:21:11.103466       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:11.103525       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:11.103649       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:11.144236       9 spark_pod_eventhandler.go:58] Pod spark-pi-c5607e749c39d210-exec-1 updated in namespace spark-operator.
I0917 13:21:11.144274       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:11.144314       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:11.144484       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:14.951818       9 spark_pod_eventhandler.go:58] Pod spark-pi-c5607e749c39d210-exec-1 updated in namespace spark-operator.
I0917 13:21:14.951881       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:14.951952       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:14.952189       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "PENDING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:21:14.952816       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"7962", FieldPath:""}): type: 'Normal' reason: 'SparkExecutorRunning' Executor spark-pi-c5607e749c39d210-exec-1 is running
I0917 13:21:15.000433       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:21:15.007030       9 sparkapp_metrics.go:256] Exporting Metrics for Executor spark-pi-c5607e749c39d210-exec-1. OldState: PENDING NewState: RUNNING
I0917 13:21:15.007076       9 metrics.go:112] Incrementing spark_app_executor_running_count with labels map[app_type:Unknown]
I0917 13:21:15.007093       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:15.007122       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:15.007222       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:19.172205       9 spark_pod_eventhandler.go:58] Pod spark-pi-c5607e749c39d210-exec-1 updated in namespace spark-operator.
I0917 13:21:19.172243       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:19.172283       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:19.172386       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:19.975344       9 spark_pod_eventhandler.go:58] Pod pi-driver updated in namespace spark-operator.
I0917 13:21:19.975374       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:19.975401       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:19.975676       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": null,
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "RUNNING"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": "2020-09-17T13:21:19Z",
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUCCEEDING"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:21:19.975879       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"7981", FieldPath:""}): type: 'Normal' reason: 'SparkDriverCompleted' Driver pi-driver completed
I0917 13:21:19.982950       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:21:19.984935       9 metrics.go:125] Decrementing spark_app_running_count with labels map[app_type:Unknown] metricVal to 0
I0917 13:21:19.984978       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:19.985020       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:19.985156       9 controller.go:773] Update the status of SparkApplication spark-operator/pi from:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": "2020-09-17T13:21:19Z",
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "SUCCEEDING"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
to:
{
  "sparkApplicationId": "spark-9dc14c7813c045529b100e3723ea0903",
  "submissionID": "5c848289-cb61-4118-b725-6d7ed8461222",
  "lastSubmissionAttemptTime": "2020-09-17T13:19:59Z",
  "terminationTime": "2020-09-17T13:21:19Z",
  "driverInfo": {
    "webUIServiceName": "pi-ui-svc",
    "webUIPort": 4040,
    "webUIAddress": "10.1.1.50:4040",
    "podName": "pi-driver"
  },
  "applicationState": {
    "state": "COMPLETED"
  },
  "executorState": {
    "spark-pi-c5607e749c39d210-exec-1": "RUNNING"
  },
  "executionAttempts": 1,
  "submissionAttempts": 1
}
I0917 13:21:19.985170       9 event.go:274] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"spark-operator", Name:"pi", UID:"773f055a-14c8-4963-ace5-34e4e051465e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"8005", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationCompleted' SparkApplication pi completed
I0917 13:21:19.996207       9 controller.go:218] SparkApplication spark-operator/pi was updated, enqueueing it
I0917 13:21:19.997570       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:19.997611       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:19.997658       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:20.001098       9 spark_pod_eventhandler.go:58] Pod spark-pi-c5607e749c39d210-exec-1 updated in namespace spark-operator.
I0917 13:21:20.001122       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:20.001140       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:20.001206       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:23.952704       9 spark_pod_eventhandler.go:58] Pod spark-pi-c5607e749c39d210-exec-1 updated in namespace spark-operator.
I0917 13:21:23.952741       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:23.952767       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:23.952935       9 controller.go:265] Ending processing key: "spark-operator/pi"
I0917 13:21:23.961083       9 spark_pod_eventhandler.go:77] Pod spark-pi-c5607e749c39d210-exec-1 deleted in namespace spark-operator.
I0917 13:21:23.961110       9 spark_pod_eventhandler.go:95] Enqueuing SparkApplication spark-operator/pi for app update processing.
I0917 13:21:23.961130       9 controller.go:258] Starting processing key: "spark-operator/pi"
I0917 13:21:23.961222       9 controller.go:265] Ending processing key: "spark-operator/pi"
doctapp picture doctapp on 17 Sep 2020

@doctapp I think there may be some issue with the mutation webhook code. Let me debug this.

jinxingwang picture jinxingwang on 17 Sep 2020
👍1

@doctapp I see the problem. you are setting allowPrivilegeEscalation: false.
And you can see, the operator is trying to patch it to /spec/securityContext from the pod spec.
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/webhook/patch.go#L540

And setting allowPrivilegeEscalation: false in the spec level securityContext is not supported.
K8s doc reference: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

If you replace allowPrivilegeEscalation: false with runAsUser: 1000, you should see a non-empty securityContext on your pod.
I think this may be a bug, because there are 2 level of securityContext we may want allow to set.
@liyinan926 what do you think? do you want me to make a fix?

jinxingwang picture jinxingwang on 17 Sep 2020

I will make a change to add a debug line here for the webhook patch result:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/7cd886c7ec449b929525c3e4d7a4b3824bf5d4b8/pkg/webhook/webhook.go#L557

jinxingwang picture jinxingwang on 17 Sep 2020
👍1

@jinxingwang appreciate if you can make a fix. Thanks!

liyinan926 picture liyinan926 on 17 Sep 2020
👍2

Would be great if the 2 levels of securityContext are supported for drivers and executors.

doctapp picture doctapp on 17 Sep 2020
👍1

Please review: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/1023

jinxingwang picture jinxingwang on 17 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hongshaoyang picture hongshaoyang  路  3Comments
jgardner04 picture jgardner04  路  5Comments
vivekjain123 picture vivekjain123  路  3Comments
AceHack picture AceHack  路  6Comments
jemega picture jemega  路  8Comments