Spark: [FEATURE REQUEST]: Support System.Decimal in CreateDataFrame API

Created on 3 Feb 2021 · 2Comments · Source: dotnet/spark

Is your feature request related to a problem? Please describe.
I'd like to be able to use the System.Decimal type when using the CreateDataFrame API. Currently when running the following:

decimal? minValue = 4.0M;
decimal? maxValue = 5.0M;
var df = sparkSession.CreateDataFrame(
    new List<GenericRow>
    {
        new GenericRow(
            new object[]
            {
                "1234",
                minValue,
                maxValue,
            }),
    },
    new StructType(
        new List<StructField>()
        {
            new StructField("id", new StringType()),
            new StructField("minValue", new DecimalType()),
            new StructField("maxValue", new DecimalType()),
        }));
df.Show();

We see the following error:
System.NotSupportedException: 'Type System.Decimal not supported yet'

This presents issues in trying to unit test code that reads decimal types from a DB.

Describe the solution you'd like
To be able to create Dataframes with Decimals without an error 😄

Describe alternatives you've considered
Alternatives for us might be to change the type being read from the DB, but this is a significant challenge - for now we are unable to unit test this code.

Additional context
N/A

enhancement triaged

Source

jgavin

Most helpful comment

Thanks @jgavin for creating this issue, we are yet to add support for System.Decimal and will get back to you about the timeline of getting this in.

Niharikadutta on 4 Feb 2021

👍3

All 2 comments

Thanks @jgavin for creating this issue, we are yet to add support for System.Decimal and will get back to you about the timeline of getting this in.

Niharikadutta on 4 Feb 2021

👍3

@Niharikadutta

Did you see that decimals were added to Apache Arrow?

This happened recently. Was that a prerequisite? Decimal types are pretty critical for us as well.