When a spark application fails at the spark-submit stage, the Spark Operator reports the application as failed, but provides no clues into what went wrong.
This is an example output from a failed submission:
applicationState": {
"errorMessage": "failed to run spark-submit for SparkApplication app-connected-stuff/connected-stuff-aggregation: exit status 1",
"state": "SUBMISSION_FAILED"
...}
Currently, the only way to debug these issues is to follow the following steps (*):
spark-submit launch commandspark-submit command obtained at step 1 A possible approach would be to capture the output of spark-submit and report it through the status stanza in the get sparkApplication command.
(*) hopefully these steps help as a workaround for other people with the same issue
Will fix this with a PR that surfaces the actual error.
Most helpful comment
Will fix this with a PR that surfaces the actual error.