Spark: Documentation for running .NET for Apache Spark on Azure Databricks

[francisco-ltech]

Is there already any documentation about how to use .Net for Apache Spark inside Azure Databricks? Do we need to wait for Azure Databricks to allow Noteboks be created with C# as a language?

[rapoth]

Yes, there are instructions to do a manual install at the moment. You can run batch jobs today on Azure Databricks.

We are currently in discussions with them to do a more deeper Notebook integration.

[francisco-ltech]

Thanks for the link. I've followed the instructions and when I try to configure spark-submit, with:

["--files","/dbfs/dotnet","--class"," org.apache.spark.deploy.DotnetRunner","/dbfs/dotnet/microsoft-spark-2.3.x-0.1.0.jar","/dbfs/dotnet/HelloSpark.zip","HelloSpark","",""]

or

["--files","/dbfs/dotnet","--class"," org.apache.spark.deploy.DotnetRunner","/dbfs/dotnet/microsoft-spark-2.3.x-0.1.0.jar","/dbfs/dotnet/HelloSpark.zip","HelloSpark"] No args as my app doesn't have any

I get the following error:

{"error_code":"INVALID_PARAMETER_VALUE","message":"Spark submit task is not allowed on existing cluster."}

Any hints?

UPDATE:

This spark-submit has to be done for a new cluster.

[imback82]

@suhsteve, can you help?

[suhsteve]

@francisco-ltech your last post has an edit/update. Does that mean you have resolved your issue?

[francisco-ltech]

@suhsteve @imback82 Yes, all good. Maybe worth amending the docs to say the spark-submit tasks needs to be for a new cluster. I'm sure any .net dev new to databricks will hit the same wall like me :-) I can send a PR for this.

Just in case anyone want to see an example of what the parameters look like:

["--files","/dbfs/mnt/dotnet","--class","org.apache.spark.deploy.DotnetRunner","/dbfs/mnt/dotnet/microsoft-spark-2.4.x-0.1.0.jar","/dbfs/mnt/dotnet/HelloSpark.zip","HelloSpark","",""]

[suhsteve]

Thanks for the update. The databricks documentation includes a note that spark-submit jobs only works on new clusters, but we can definitely amend our docs also and include a note.

Regarding your example parameters, can you try also try to run it with:
["--class","org.apache.spark.deploy.DotnetRunner","/dbfs/mnt/dotnet/microsoft-spark-2.4.x-0.1.0.jar","/dbfs/mnt/dotnet/HelloSpark.zip","HelloSpark"]

You shouldn't need --files dotnet if you published your application using dotnet publish -r <runtime> ... as this will include the .NET Core runtime and libraries with your application and publish your app as an executable, ie the HelloSpark executable file in your applications build output directory.

[francisco-ltech]

Yes, that also works. Thanks :-)

On a separate question: Is there a list of the APIs that are not yet available? I read SparkContext.Parallelize is one of them but wondering if there is like an "official" tracking list somewhere?

[rapoth]

CC: @imback82 for comment on the list of APIs. We do not support RDDs at the moment (per advice from Reynold Xin from Databricks) and hence Parallelize is not opened up.

On a related note, I've updated the Running on Databricks documentation - Do you have any thoughts? Here's the https://github.com/dotnet/spark/pull/94 where I brought in these changes.

[francisco-ltech]

@rapoth Thanks for updating the docs, I think it reads much clearer now.