Spark: [BUG]: Error while trying to enumerate a DataFrame containing a column of type date: The method or operation is not implemented.

[zwitbaum]

Describe the bug
An error occurred while trying to enumerate a DataFrame containing a column of type date.

To Reproduce
Steps to reproduce the behavior:

public void Test()
{
  var dataFrame1 = spark.Session.Sql("SELECT * FROM VALUES ('2020-1-1'), ('2020-1-2') AS (DateAsString)");
  dataFrame1.Show();
  dataFrame1.PrintSchema();
  var rows1 = dataFrame1.Collect().Count(); // returns 2

  var dataFrame2 = spark.Session.Sql("SELECT * FROM VALUES CAST('2020-1-1' AS date), CAST('2020-1-2' AS date) AS (DateAsDate)");
  dataFrame2.Show();
  dataFrame2.PrintSchema();
  var rows2 = dataFrame2.Collect().Count(); // failed with "System.NotImplementedException" {"The method or operation is not implemented."}
}

Command-Line output:
+------------+
|DateAsString|
+------------+
| 2020-1-1|
| 2020-1-2|
+------------+

root
|-- DateAsString: string (nullable = false)

+----------+
|DateAsDate|
+----------+
|2020-01-01|
|2020-01-02|
+----------+

root
|-- DateAsDate: date (nullable = true)

Exception
"System.NotImplementedException: The method or operation is not implemented.
at Microsoft.Spark.Sql.Row.Convert()
at Microsoft.Spark.Sql.Row..ctor(Object[] values, StructType schema)
at Microsoft.Spark.Sql.RowConstructor.GetRow()
at Microsoft.Spark.Sql.RowCollector.Collect(ISocketWrapper socket)+MoveNext()
at Microsoft.Spark.Sql.DataFrame.GetRows(String funcName)+MoveNext()
at System.Linq.Enumerable.CountTSource
at MyTest.Test() in ..."

Expected behavior
A DataFrame containing columns of type date can be enumerated.

Additional context
Tested with a latest version (0.7.0) of the .net spark library.

[michael-damatov]

When some data types (like _date_ or _map_) are not yet supported, it's ok to throw an exception. However, in this case I'm not even able to count the rows regardless of their column types.

[imback82]

@zwitbaum yes, this is known issue: #26. We will address this in upcoming releases.

@michael-damatov can't you just use this function to get the row count? https://github.com/dotnet/spark/blob/5e9c08b430b4bc56b5f42252c4b73437377afaed/src/csharp/Microsoft.Spark/Sql/DataFrame.cs#L723-L727

[michael-damatov]

@imback82 calling DataFrame.Count works. Is there a way not to throw exceptions from the Row constructor, just to allow iterating over rows?

[imback82]

@michael-damatov currently no, but we are working to support all the complex types in upcoming releases, and we welcome contributions too!

[imback82]

cc @elvaliuliuliu

[elvaliuliuliu]

I am working on this and it should be supported in the coming releases. Thanks!

[ravshMSFT]

I am getting same error when I do dataframe.collect().ToList() though my dataframe does not contains any complex types. It has just string, int and decimals.

Any idea why it is failing for me ?

System.NotImplementedException: The method or operation is not implemented.
at Microsoft.Spark.Sql.Row.Convert()
at Microsoft.Spark.Sql.RowConstructor.GetRow()
at Microsoft.Spark.Sql.RowCollector.Collect(ISocketWrapper socket)+MoveNext()
at Microsoft.Spark.Sql.DataFrame.GetRows(String funcName)+MoveNext()
at System.Collections.Generic.List1.AddEnumerable(IEnumerable1 enumerable)
at System.Linq.Enumerable.ToListTSource

[imback82]

@ravshMSFT, can you paste your schema for this dataframe? (datafarme.PrintSchema())

[ravshMSFT]

Here is the schema from dataframe
root
|-- DataAreaId: string (nullable = true)
|-- JobId: string (nullable = true)
|-- WrkCtrDigitalId: string (nullable = true)
|-- ActualQty: long (nullable = false)
|-- RequestedQty: decimal(8,2) (nullable = true)
|-- ExpectedQty: double (nullable = true)
|-- ProductionRateDeviationThreshold: string (nullable = true)

[elvaliuliuliu]

Can you try with Spark.NET v0.9.0? I think this should work with the new version.

[ravshMSFT]

code from 0.5.0. Seems decimal type also not implemented.

private void Convert()
{
foreach (StructField field in Schema.Fields)
{
if (field.DataType is ArrayType)
{
throw new NotImplementedException();
}
if (field.DataType is MapType)
{
throw new NotImplementedException();
}
_if (field.DataType is DecimalType)
{
throw new NotImplementedException();
}_
if (field.DataType is DateType)
{
throw new NotImplementedException();
}
}
}

[ravshMSFT]

Upgraded to 0.9.0 and this issue is resolved.