Spark: Databricks Deployment: Init Script Failure

[bamurtaugh]

I have been following the cloud deployment guide (https://github.com/dotnet/spark/blob/master/deployment/README.md) to deploy my app to Databricks. I have installed db-init and install-worker to my local machine, and I have modified db-init to the appropriate version of the Spark Worker release (the same version I got working when I deployed to HDInsight).

This is the only part of db-init I changed:
DOTNET_SPARK_RELEASE=https://github.com/dotnet/spark/releases/download/v0.1.0/Microsoft.Spark.Worker.netcoreapp2.1.linux-x64-0.1.0.tar.gz

I've uploaded db-init and install-worker to dbfs. When I create a cluster and add the db-init script action, I get the following error and the cluster cannot be created:
Cluster terminated. Reason: Init Script Failure
Init script dbfs:/spark-dotnet/db-init.sh failed: Script exit status is non-zero

I opened db-init in Atom and changed the line endings to LF (Unix-style) and reuploaded to dbfs. I've also tried modifying the DOTNET_SPARK_RELEASE value in db-init with different versions of the Microsoft Worker release, and then re-uploaded db-init to dbfs, but I always get the same error that the cluster creation failed.

Any feedback about how to approach this issue would be great!

[rapoth]

@suhsteve - Any suggestions?

[bamurtaugh]

Update: I was able to get the logs from db-init failing on my cluster, and it appears that it's an issue with the line endings in the install-worker script:

/dbfs/spark-dotnet/install-worker.sh: line 2: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 24: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 25: set: +
: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
/dbfs/spark-dotnet/install-worker.sh: line 26: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 29: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 32: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 35: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 38: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 42: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 45: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 49: $'\r': command not found
/dbfs/spark-dotnet/install-worker.sh: line 65: syntax error near unexpected token elif' /dbfs/spark-dotnet/install-worker.sh: line 65:elif [ "${CLOUD_PROVIDER,,}" = "azure" ]; then

I had changed my line endings to Unix-style for db-init in Atom but not for install-worker.sh (I thought install-worker was okay because I was able to use it as-is in HDInsight deployment). I converted the line endings in Atom for install-worker and reuploaded it to the cluster.

Cluster creation was now successful!

[bamurtaugh]

Closing this issue since I was able to get a cluster running and use spark-submit to successfully run my app.

[rapoth]

Awesome, thanks for the update!