Spark: [FEATURE REQUEST]: Support createDataFrame
Is your feature request related to a problem? Please describe.
Spark.NET does not support createDataFrame API
Describe the solution you'd like
Adding the createDataFrame API would be great since it allows for faster experimentation and playing with the library.
rapoth
All 2 comments
My application is that I need to append a C# List into a delta Lake table. With Spark 2.4 I can do this but I first need to convert this list into a DataFrame. From what I can see there is currently no API to do this using .NET-spark. I would need to either save this to a temp file (Parquet, JSON etc..) or stream into something such as Kafka or EventHub.
With PySpark I can convert a Python list using the createDataFrame API. I feel this would be the best solution to my problem if this API was available.
oscarwarnershell
on 9 Aug 2019
Closed by #338. This is released in 0.8.0.
imback82
on 25 Jan 2020
Related issues
cartermp
路
3Comments
fabiogouw
路
9Comments
RoboBuffalo
路
11Comments
bamurtaugh
路
5Comments
bamurtaugh
路
4Comments
Most helpful comment
My application is that I need to append a C# List into a delta Lake table. With Spark 2.4 I can do this but I first need to convert this list into a DataFrame. From what I can see there is currently no API to do this using .NET-spark. I would need to either save this to a temp file (Parquet, JSON etc..) or stream into something such as Kafka or EventHub.
With PySpark I can convert a Python list using the createDataFrame API. I feel this would be the best solution to my problem if this API was available.