Onednn: Does INT8 scale in attribute belong to code or data in the computation?

[CaoZhongZ]

https://github.com/intel/mkl-dnn/blob/4ade75c386c8bea56bd60529b021d8b853d62617/src/common/primitive_hashing.hpp#L90

Does JIT embed scale as code or just reference it as a array? If it reference it as a array, we shouldn't hash the content of it right? Should it be a buffer which can be set and get?

After we have the primitive cache, this contents will be accessed over and over, it'll result some noticeable overhead when C is significant.

[rsdubtso]

The scales are copied and stored inside the primitive.

[CaoZhongZ]

Yes, I mean if C is large, it'll be costly to do so. Is it necessary to do so (faster JIT code?), or we could just pass a scale buffer in every time. In that case, we also don't have to hash the contents and save another overhead.

[emfomenk]

The scales are copied and stored inside the primitive.

A little more of background. Scales are part of primitive descriptor, which cannot be modified afterwards. When a primitive is executed, it takes the scales from the primitive descriptor. Hence, the scales indeed should be hashed.

Implementation-wise, what matter for the jitted kernel is how many scales we have (none, common per tensor, per channel). In the latter case we don't embed the scales themselves in the code. So in theory we can ask to pass us the scales at run time, not at creation time.
However, in case of reorders, when the scale is common per tensor, we check if this scale is 1. or something else. In the latter case we insert mulps instruction, which don't do in the former case. So from that perspective the jitted code does depend on the scaling factors.

[emfomenk]

or we could just pass a scale buffer in every time

Currently, you can't. The scales are copied, and further modifications to the original scales don't affect the primitive itself (since it has a copy in its primitive descriptor).
We are considering adding a special value DNNL_RUNTIME_F32_VAL that user can pass to the attribute scales to say the library that actual scales should be pass at execution time. Something like this:

// pd creation
attr.set_output_scale(mask, {DNNL_RUNTIME_F32_VAL});
convolution_forward::primtive_desc cpd(..., attr);
convolution_forward conv(pd);

// execution
conv.execute(...,
    {DNNL_ARG_ATTR_OUTPUT_SCALE, memory({{size}, f32, tag::x}, engine, runtime_scales_ptr)},
);

[pinzhenx]

We are considering adding a special value DNNL_RUNTIME_F32_VAL ...

Which version is this feature going to be shipped with, probably?

[emfomenk]

Most likely DNNL v1.3 (somewhere in Q1'2020). But please take this with a grain of salt.
We are now in the very early stage of PoC that would have this feature. The goal of the PoC to make sure that everything works fine. So the actual timeline depends on:

• How well the PoC goes and how long it takes;
• Other urgent requests we might get;
• Priorities for the other things;

[pinzhenx]

@emfomenk Thanks for your explanation.

[CaoZhongZ]

However, in case of reorders, when the scale is common per tensor, we check if this scale is 1. or something else. In the latter case we insert mulps instruction, which don't do in the former case. So from that perspective the jitted code does depend on the scaling factors.

What's the possibility that a scale equals one? Reorder is a memory intensive operation, which might allow a little more computation to squeeze in?

I recently review INT8 integration and I think we should raise this question long time ago. Anyway, pass it in runtime should be a good solution.

[emfomenk]

What's the possibility that a scale equals one? Reorder is a memory intensive operation, which might allow a little more computation to squeeze in?

For reorders scales = 1 is pretty often situation. Scales is not equal to 1 in the following case:

• Reorder is a part of concat with input scale != 1;
• Reorder is actually q10n or deq10n between f32 <-> i8, or rescaling between i8 <-> i8.

But I do agree, that this not something very important. We can always multiply by 1.f and that should not affect performance much. Or we can just do a run-time dispatching between the kernel the does scaling and the kernel that does not do it.

[emfomenk]

As the original question answered I am closing the issue. The run-time attributes hopefully would become a reality soon which will also resolve the concern about performance (even though to me that is very minor issue compare with overall inconvenience of copying the scales inside the primitive / kernel).
Please feel free to re-open this issue or open a new one if you have any further questions / comments.