Glow: [New operator] ReplaceNaN

[artemrakhov-glow]

We need to support ReplaceNaN operator from Caffe2. Spec can be found here:
https://caffe2.ai/docs/operators-catalogue.html#replacenan

[SplitInfinity]

I want to implement this and I'm thinking of doing immediate lowering using CmpEq and Select nodes. But how do I portably represent NaN? I know there is some IEEE-defined bit pattern that indicates NaN, but I can't find a nice helper function anywhere.

[SplitInfinity]

Hmm, will this work: https://en.cppreference.com/w/cpp/numeric/math/NAN?

[artemrakhov-glow]

The link that you provided works, and Caffe2 uses std::isnan. But using it would require creating this new operator for every single backend.

However, we can use a trick that x == x is true for every floating number except NaN. So compare input to itself, elements that are not equal are NaNs. And of course, this needs a test.

[nadavrot]

@artemrakhov Ah, interesting. We need to ensure that the optimizer does not replace "x==x" with "true". I wonder if isNaN is special enough to be a dedicated node. I wonder if this operation would be translated to NOP on some hardware accelerators that never produce NaN. I guess that the fact that we need to replace the value with some other scalar forces us to implement the compare-select semantics. This is a hard one. What do other people think?

[bertmaher]

I'm in favor of making it a special node. NaN semantics are pretty weird even on standard hardware. I think std::isnan is probably the way to go.

[rdzhabarov]

Some additions.

This needs ONNX loader as well.

[SplitInfinity]

@rdzhabarov

Is this officially part of the ONNX spec? Is there a standard way to represent this operator in a .onnxtxt?

[rdzhabarov]

Is this officially part of the ONNX spec? Is there a standard way to represent this operator in a .onnxtxt?

As discussed offline, that's our custom extension.