Spark-on-k8s-operator: Improve error reporting of failed applications

Created on 31 Jan 2019  路  1Comment  路  Source: GoogleCloudPlatform/spark-on-k8s-operator

When a spark application fails at the spark-submit stage, the Spark Operator reports the application as failed, but provides no clues into what went wrong.
This is an example output from a failed submission:

applicationState": {
            "errorMessage": "failed to run spark-submit for SparkApplication app-connected-stuff/connected-stuff-aggregation: exit status 1",
            "state": "SUBMISSION_FAILED"
...}

Currently, the only way to debug these issues is to follow the following steps (*):

  1. access the SparkOperator logs
  2. find the spark-submit launch command
  3. start a shell to the SparkOperator pod
  4. manually launch the complete spark-submit command obtained at step 1
  5. figure out what was wrong, fix, rinse, repeat

A possible approach would be to capture the output of spark-submit and report it through the status stanza in the get sparkApplication command.

(*) hopefully these steps help as a workaround for other people with the same issue

Most helpful comment

Will fix this with a PR that surfaces the actual error.

>All comments

Will fix this with a PR that surfaces the actual error.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

andrewgdavis picture andrewgdavis  路  5Comments

mjschmidt picture mjschmidt  路  7Comments

AceHack picture AceHack  路  6Comments

liyinan926 picture liyinan926  路  4Comments

Venkatesh89-Github picture Venkatesh89-Github  路  8Comments