Spark: [BUG]: .Net Interactive: UDF use deprecated BinaryFormatter, fail in VS Code

[oefe]

Describe the bug
UDFs are serialized using BinaryFormatter, which is deprecated for security reasons.

In environments where BinaryFormatter is already disabled, this causes UDFs to fail with a NotSupportedException.

To Reproduce

Steps to reproduce the behavior:

1. Use Visual Studio Code with the .Net Interactive Notebooks extension
2. Follow the steps in UDFs in .NET Interactive
3. Executing cell #3 (which contains the definition of the UDF) fails with:

Error: System.NotSupportedException: BinaryFormatter serialization and deserialization are disabled within this application. See https://aka.ms/binaryformatter for more information.
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph)
at Microsoft.Spark.Utils.CommandSerDe.Serialize(Delegate func, SerializedMode deserializerMode, SerializedMode serializerMode)
at Microsoft.Spark.Sql.Functions.CreateUdf(String name, Delegate execute, PythonEvalType evalType, String returnType)
at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute, PythonEvalType evalType)
at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute)
at Microsoft.Spark.Sql.Functions.Udf[T,TResult](Func`2 udf)
at Submission#11.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)

Expected behavior

• Should be able to use UDFs
• The example in UDFs in .NET Interactive should work
• UDFs should not use BinaryFormatter

Screenshots
NA

Desktop (please complete the following information):

• OS: macOS 10.15.7
• .NET 5 (5.0.100)
• Apache Spark 3.0.1
• .NET for Apache Spark v1.0.0 (netcoreapp3.1, macOS)
  
  
  
  • VS Code:
  
  
  • Version: 1.53.0-insider
  
  
  • Commit: 96b426ef1da0f0a51ce9ef1931cd1fd2322547e4
  
  
  • Date: 2020-12-14T05:49:45.285Z
  
  
  • Electron: 11.0.3
  
  
  • Chrome: 87.0.4280.67
  
  
  • Node.js: 12.18.3
  
  
  • V8: 8.7.220.25-electron.0
  
  
  • OS: Darwin x64 19.6.0
  
  
  • .Net Interactive Notebooks v1.0.160901
  
  

Additional context
NA

[oefe]

The same example also fails in Jupyter, but for a different reason: #796

[imback82]

Thanks @oefe for reporting this. We will move away from BinaryFormatter from upcoming releases.

[imback82]

cc @suhsteve @jonsequitur (FYI)

[Frassle]

Not sure what the dotnet plan for replacing BinaryFormatter here is, but I was wanting to use net spark in F# notebooks (where UDFs have never been supported), plus I had a couple of other similar problems that all looked solvable if dotnet had something like pythons cloudpickle. So I wrote one, https://github.com/Ibasa/Pikala. Forked spark to use it and can now run F# UDFs from notebooks.