Onnxruntime: "Float16" data type are not support in C# (Which mean it will impact to use the FP16 model in Onnxruntime)

[NWH1111]

Is your feature request related to a problem? Please describe.
When I tried to test with Float16 model, it will run with error -
"Unexpected input data type. Actual: (class onnxruntime::NonOnnxType float) , expected: (class onnxruntime::NonOnnxType union onnxruntime::MLFloat16)". cause I cannot find any datatype to fulfill MLFloat16 requirement (so I put the float as input).

It seems like there's lack of float16 support in onnxruntime. Can anybody know how to pass data to "float16" onnx models in C#? Thanks.

Here is the quick sample in my code:
var container = new List();
container.Add(NamedOnnxValue.CreateFromTensor <float> (inputName, inputTensor));
using (var results = session.Run(container)) --> it will pop up an error "Unexpected input data type" like above

System information

• Microsoft.ML.OnnxRuntime.Gpu version (0.3.1):
• Windows 10 19H1 with RTX2080

Describe the solution you'd like
DataType can be used for Float16 model and it will auto convert from float32 --> float16

Describe alternatives you've considered
Provide any datatype can fulfill Float16 model

[raymondxyang]

Hi thanks for trying our C# support. If I remember correctly we have discussed this item before in the meeting @jignparm would you mind following up this issue?

[tannergooding]

The framework side of this is being tracked by https://github.com/dotnet/corefx/issues/25702 and is being prototyped under https://github.com/dotnet/corefxlab/issues/2635.

There are also discussions to ensure that it gets the appropriate language support in C# (and potentially other languages like F#).

[linkerzhang]

ORT will also need to expose the support via C# API, @faxu for tracking it.

[jignparm]

The ORT C# API maps several tensor native types to C# system types (e.g. Int64 -> long, see below). Currently Bloat16 is not implemented. Ideal solution is to have Bfloat16 implemented as a new type in C# as @tannergooding mentioned above.

https://github.com/microsoft/onnxruntime/blob/master/csharp/src/Microsoft.ML.OnnxRuntime/NamedOnnxValue.cs#L327

[tannergooding]

@jignparm, do you need both bfloat16 (Google's format) and float16 (the IEEE half-precision format)?

We have proposals to support both currently and will likely end up implementing both, but I just want to make sure I have it tracked correctly.

[jignparm]

The TensorProto definition contains both Float16 as well as BFloat16 (see below). We would need to support both the formats in the C# API. There are currently a handful of Float16 models in the test suite (half-precision) which cannot be scored in C#, but are fine in native C++. Is there a timeline for the proposal? There seems to be a growing interest in 16-bit models.

https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/protobuf/onnx-ml.proto#L325

// IEEE754 half-precision floating-point format (16 bits wide).
// This format has 1 sign bit, 5 exponent bits, and 10 mantissa bits.
FLOAT16 = 10;

// Non-IEEE floating-point format based on IEEE754 single-precision
// floating-point number truncated to 16 bits.
// This format has 1 sign bit, 8 exponent bits, and 7 mantissa bits.
BFLOAT16 = 16;

[Voxelworld]

I observed a similar problem when using the Python API after converting my model to float16 (using the ONNX converter script):

RuntimeError: Numpy_type 23 can't be converted to MLDataType.

Is this related to the C# fix or should I open a new issue?

Tested with the CPU version of the onnxruntime 0.4.0 using pip install onnxruntime.

[tannergooding]

Is there a timeline for the proposal?

I believe the current target is .NET 5, but it also depends largely on prioritization and scheduling.

[stale[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[pedramrezaei]

What is the latest on this?

[tannergooding]

.NET 5 is shipping System.Half as an interchange type: https://devblogs.microsoft.com/dotnet/introducing-the-half-type/

This means you can parse/format strings, do basic comparison/equality checks, and convert to/from float and double. Other operations, such as addition, subtraction, and language literals aren't available.

If language support is needed, I'd recommend opening an issue on dotnet/csharplang requesting it and detailing the requirements, reasons, and scenarios.

[pedramrezaei]

Thanks. TBH, whether .NET supports float16 or not is orthogonal to the issue here. ORT supports optimizing BERT like models to use FP16. However, the library provides no obvious way of consuming these models. Also, .NET 5 is not released and not an option at this time. Can the C# ORT internally convert FP32 to an equivalent of FP16?

[jdluzen]

I agree with pedramrezaei. Language support for 16 bit floats would be great, but is not strictly needed. A perfectly acceptable workaround would be to somehow manage 16 bit inputs/outputs manually. We just need a way to consume these models.

[yuslepukhin]

Depends on https://github.com/microsoft/onnxruntime/issues/5679

[yuslepukhin]

I am curious as to what data types people are currently using in C# to deal with Float16 and BFloat16? I will introduce a struct type into C# API and it will serve well to feed and fetch such data. However, if you use your own type, you may need to copy which is not the best thing in the world. Is it really necessary to support some arbitrary type arrays that are size of ushort and blittable so you can feed your arrays. This comes at a cost of safety since the code would have to believe you that you are doing things correctly.