Spark: [Documentation]: Update for 1.0 Release

[Niharikadutta]

As we gear up towards our 1.0 release, documentation needs to be added about the following:

New docs:

How to's

• [x] Connecting to Remote Storage from your Local Machine
  
  
  
  • [x] Connect to ADLS gen2 account locally (#337) - PR link
  
  
  • [x] Connect to Azure Blobs (WASB) account locally (#337) - PR link
  
  
• [x] Connecting to Data Sources
  
  
  
  • [x] SQL Server - PR link
  
  
  • [x] EventHub - PR link
  
  
  • [x] MongoDB - PR link
  
  
• [x] Use .NET for Apache Spark locally using Jupyter notebooks (#529) - PR link
• [x] Use .NET for Apache Spark locally using VS Code - PR link
• [x] User-defined Functions in .NET for Apache Spark
  
  
  
  • [x] Invoking Scala/Java User-defined Functions from Spark apps written in C# - PR link
  
  
  • [x] Invoking C# User-defined Functions from Spark apps written in Scale/Java/Python (import this) - PR link
  
  
• [x] Using .NET for Spark on Azure Synapse Analytics - Update links after how to guides published
  
  
  
  • [x] Using .NET Notebooks (including where to find documentation pertaining to .NET Interactive) - PR link
  
  
  • [x] Installing Nuget Packages in Notebooks - PR link
  
  

Dotnet-interactive guide/good to know's

• [x] Dotnet interactive serialization issue (#619) - PR link
• [x] Running UDFs in notebooks - PR link

Migration guide - #721

• [x] Document moving Microsoft.Spark.Experimental to Microsoft.Spark. - #691
• [x] Known issues: e.g.) grouped map UDF for Spark 3.0
• [x] .NET Delta/Hyperspace version | OSS Delta/Hyperspace version | Spark versions matrix
• [x] Dropping support for .NET Framework
• [x] Show Scala version in microsoft-spark*.jar filename

Existing docs

• [x] Update this documentation to reflect how to zip binaries for .NET for Spark batch submission. - PR link
• [x] Update FAQs to include how ArrayList and HashTable cannot be supported as UDF return type #684 - PR link - Update links after how to guides published
• [x] Verify (and simplify) this - PR link
• [x] Update this for Submitting batch jobs (with additional dependencies) - PR link
• [x] Verify all public facing documentation
  
  
  
  • [x] GitHub documentation
  
  
  • [x] .NET Website (https://dot.net/spark) - PR link
  
  
  • [x] Azure Synapse Analytics - PR link - Update links after how to guides published
  
  

[rapoth]

@luisquintanilla @mrys @JeremyLikness Do you have any other topics we should cover (based on what you heard from users)?

[Niharikadutta]

For "Using .NET for Spark on Azure Synapse Analytics - Submitting batch jobs", should we just update this documentation instead of creating a new one?

[rapoth]

Yes, please! That's a great idea! Can you also update your issue with any existing links to docs that already exist and that need to be updated?

[luisquintanilla]

Not sure exactly where this would fit. Maybe it can be a standalone article, but basically, how can you make use of Scala libraries that run on top of Apache Spark within .NET for Apache Spark?

Some Examples:

• MMLSpark.
• TransmogrifAI

At the moment there are no .NET bindings for these libraries. Can I still use them with .NET for Spark? If so, how?

[luisquintanilla]

@Niharikadutta

Just wanted to clarify what is meant for this work item. Does this mean, using .NET Interactive in VS Code with .NET for Apache Spark?

Use .NET for Apache Spark locally using VS Code - @luisquintanilla working on this

[imback82]

Linking #293 as migration guide

[imback82]

Can we also capture the followings in the descrption?

1. Known issues: e.g.) grouped map UDF for Spark 3.0

2. .NET Delta/Hyperspace version | OSS Delta/Hyperspace version | Spark versions matrix

[imback82]

Can we also capture dropping support for .NET Framework?

[Niharikadutta]

@luisquintanilla As discussed offline, what Use .NET for Apache Spark locally using VS Code this work item means is using using .NET Interactive in VS Code with .NET for Apache Spark using the .NET Interactive Notebooks extension.
And regarding your other question about using external Scala libraries through .NET for Apache Spark, I am working on that and will update the thread as soon as I have something concrete. Thanks!

[rapoth]

@indy-3rdman Do you have any feedback on this thread (specifically, if any additional topics need to be covered OR feedback on existing PRs which are linked to from the parent thread)? Thank you!

[rapoth]

@Niharikadutta Do you think we can close this issue?

[Niharikadutta]

@Niharikadutta Do you think we can close this issue?

Yes we can close this.