Spark: Using with .NET Interactive notebooks locally

Created on 23 May 2020 · 6Comments · Source: dotnet/spark

I'd ideally like to submit jobs to my Spark cluster (local or remote) via a Jupyter notebook. It turns out that there's a recent update to .NET Interactive to allow me to use their C# REPL within Jupyter, but I'm unable to find any documentation indicating how to use this with Spark.NET.

Both Synapse and Databricks offer support to use notebooks online with Spark, so I'm looking for guidance on how to do the same locally. Thank you!

Source

WhitWaldo

👍1

Most helpful comment

Yes, this is doable.

First, run a spark-submit with a debug mode:

> spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner microsoft-spark-2.4.x-0.10.0.jar debug
20/05/22 21:01:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/05/22 21:01:06 INFO DotnetRunner: Starting DotnetBackend with .
20/05/22 21:01:06 INFO DotnetRunner: Port number used by DotnetBackend is 5567
***********************************************************************
* .NET Backend running debug mode. Press enter to exit *
***********************************************************************

Now, open up the Jupyter notebook with dotnet-interactive kernel (C#), install Microsoft.Spark Nuget with #r "nuget: Microsoft.Spark", and start writing your application:

The ability to write UDF in the notebook will be introduced with #515 and #517 and once they are both merged, we will write up a guide on how to use this locally.

Thanks!

imback82 on 23 May 2020

👍4

All 6 comments

Yes, this is doable.

First, run a spark-submit with a debug mode:

> spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner microsoft-spark-2.4.x-0.10.0.jar debug
20/05/22 21:01:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/05/22 21:01:06 INFO DotnetRunner: Starting DotnetBackend with .
20/05/22 21:01:06 INFO DotnetRunner: Port number used by DotnetBackend is 5567
***********************************************************************
* .NET Backend running debug mode. Press enter to exit *
***********************************************************************

Now, open up the Jupyter notebook with dotnet-interactive kernel (C#), install Microsoft.Spark Nuget with #r "nuget: Microsoft.Spark", and start writing your application:

The ability to write UDF in the notebook will be introduced with #515 and #517 and once they are both merged, we will write up a guide on how to use this locally.

Thanks!

imback82 on 23 May 2020

👍4

cc: @jonsequitur (just FYI)

imback82 on 23 May 2020

@WhitWaldo We believe this issue to be resolved, do you need any further help in running your spark jobs via Jupyter?

Niharikadutta on 27 Aug 2020

Hi, we are going to close this issue as it has been inactive for a while. Please feel free to re-open it if the issue persists and/or there are any new updates. Thank you!

Niharikadutta on 9 Sep 2020

Reopening to follow up on the documentation.

imback82 on 9 Sep 2020

@WhitWaldo We recently wrote an article to document the steps to use .NET interactive notebooks locally using .NET for Apache Spark, please feel free to check it out here.
Closing this issue.

Niharikadutta on 14 Oct 2020

Was this page helpful?

0 / 5 - 0 ratings