Spark-on-k8s-operator: Looks like the spark service account is not created

[mjschmidt]

Should the spark helm chart be creating the service account for spark?

The logs show otherwise, but if not some simple documentation would be nice:

kubectl get sparkapplications spark-pi -o=yaml | grep service


      {"apiVersion":"sparkoperator.k8s.io/v1beta1","kind":"SparkApplication","metadata":{"annotations":{},"name":"spark-pi","namespace":"default"},"spec":{"driver":{"coreLimit":"200m","cores":0.1,"labels":{"version":"2.4.0"},"memory":"512m","serviceAccount":"spark","volumeMounts":[{"mountPath":"/tmp","name":"test-volume"}]},"executor":{"cores":1,"instances":1,"labels":{"version":"2.4.0"},"memory":"512m","volumeMounts":[{"mountPath":"/tmp","name":"test-volume"}]},"image":"gcr.io/spark-operator/spark:v2.4.0","imagePullPolicy":"Always","mainApplicationFile":"local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar","mainClass":"org.apache.spark.examples.SparkPi","mode":"cluster","restartPolicy":{"type":"Never"},"sparkVersion":"2.4.0","type":"Scala","volumes":[{"hostPath":{"path":"/tmp","type":"Directory"},"name":"test-volume"}]}}
    serviceAccount: spark
      Message: Forbidden!Configured service account doesn't have access. Service account
      up service account default/spark: serviceaccount \"spark\" not found.\n\tat

 kubectl describe sparkapplication spark-pi


Events:
  Type     Reason                            Age   From            Message
  ----     ------                            ----  ----            -------
  Normal   SparkApplicationAdded             9m9s  spark-operator  SparkApplication spark-pi was added, enqueuing it for submission
  Warning  SparkApplicationSubmissionFailed  9m7s  spark-operator  failed to submit SparkApplication spark-pi: failed to run spark-submit for SparkApplication default/spark-pi: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.43.0.1/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-driver" is forbidden: error looking up service account default/spark: serviceaccount "spark" not found.
           at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
           at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:407)
           at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
           at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
           at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:226)
           at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:769)
           at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:356)
           at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:141)
           at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:140)
           at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
           at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:140)
           at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:250)
           at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:241)
           at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
           at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
           at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
           at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
           at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
           at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
           at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
           at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
           at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
           at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[liyinan926]

What's the command you used to install the Helm chart? There's a config property sparkJobNamespace that you need to specify to enable auto creation of the service account. The value of this property is the namespace in which you want to run your SparkApplications.

[mjschmidt]

this works:
helm install --name=spark incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=spark-jobs --set serviceAccounts.spark.name=spark

this doesn't:
helm install --name=spark incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=spark-jobs I believe we can update the values file to just set serviceAccounts.spark.name=spark. I can do a pull request later, but its a simple one liner if you want to fix it now.

The issue is that if you leave serviceAccounts.spark.name as a blank value in your values file the default service account name is spark-spark, but the examples point to a service account name of just spark

[mjschmidt]

@liyinan926 I have a simple pull request in. Feel free to implement the same pull and I will delete mine.

[yuchaoran2011]

@liyinan926 This issue can now be closed. I had some exchanges with @mjschmidt under his PR and we agreed to keep the current defaults.

[liyinan926]

Thanks @yuchaoran2011 and @mjschmidt.

[mjschmidt]

violent exchanges hahaha actually @yuchaoran2011 after using the spark operator more it totally makes sense and I agree with you now.

[yuchaoran2011]

@mjschmidt lol awesome!