Onednn: Question: Is convolution with signed 8bit source supported?

[SafeteeWoW]

I want to do convolution based on the following format:
source(s8), weight(s8), bias(s32) ---> dest(s32)

However, it seems source only supports unsigned 8bit right now.

I am following the example of simple_net_int8.cpp,
but when the memory format of conv_src_md from u8 to s8 does not give me correct output.

Is signed 8 bit source convolution supported? If yes, can you support a sample code?

[emfomenk]

Hi @SafeteeWoW,

Can you please try using the latest (master) version of Intel MKL-DNN?
We recently extended support with s8 (*) s8 --> s32 convolutions.

Here are few implementations:
https://github.com/intel/mkl-dnn/blob/master/src/cpu/cpu_engine.cpp#L152
https://github.com/intel/mkl-dnn/blob/master/src/cpu/cpu_engine.cpp#L169

[SafeteeWoW]

@emfomenk
Thanks for your reply.

https://gist.github.com/SafeteeWoW/d5a105041e88ba47b47bf2990d26ebc2

In the Gist above, I run signed int8 convolution on:
src : 1x3x64x64, all values filled with 1.
weight: 8x3x5x5, all values filled with 1.
bias: 8, all values filled with 0.

I expect the first value of the output (conv_dst@(0,0,0,0)) should be 75, which is true when I set the memory of src to be memory::data_type::u8. However, I get 0 when I set the memory of src to be memory::date_type::s8.

I am using the latest version on the master branch right now (commit hash 1687299).
My CPU is Xeon E5-2667.
My compiler is gcc 7.3.0
My Cmake is 3.12.4
My OS is Linux CentOS 7.3 (Kernel 3.10.0-514.el7.86_64)
I compiled MKL-DNN with default configuration with small MKL library https://github.com/intel/mkl-dnn/releases/download/v0.17.2/mklml_lnx_2019.0.1.20181227.tgz
I didn't set any MKL related environment variables during compilation of MKL-DNN or running the program in the Gist above.

[emfomenk]

Thanks for a small reproducer!
I was able to reproduce the issue.
Let me take a look into this and come to you...

[emfomenk]

Oh, here is the thing...

This is intentional.
If you change weights fill from 1 to 2 you would get the same (expected) results: 150.

The reason is that during weights reorder we divide all the data by two to overcome possible overflows. In the real world examples (e.g. inference using Intel Caffe that brings minor accuracy drop, but still acceptable). If we would not do that, the accuracy might be much bigger.

Executive summary. The int8 convolutions (on <= Skylake) are defined as:

u8 (*) s8 case:
    dst_s32 <- (s32_with_interim_s16) src_u8 (*) wei_s8

s8 (*) s8 case:
    dst_s32 <- 2 * (s32) ((src_s8 + 128) (*) (wei_s8 / 2)) - 128 (*) wei_s8

In your example: wei_s8 = {1, ..., 1}, hence wei_s8 / 2 = {0, ..., 0}, hence the result is 0.

---

Let me try to elaborate why Intel MKL-DNN has such weird behavior (be ready, the explanation is long).
For u8 (*) s8 -> s32 convolution the real chain of operations on Skylake is:

L01     dst_s32 = 0;
L02     for (k = 0 .. dst_size / 2) {
L03        dst_s32 += (s32)(
L04             (s16)src_u8[2k + 0] * wei_s8[2k + 0]
L05             +
L06             (s16)src_u8[2k + 1] * wei_s8[2k + 1]
L07         );
L08     }

The intermediate casting to (s16) is caused by ISA. See VPMADDUBSW instruction (some comments can be found in 1).

Note, that in addition in line (L05) might overflow.
For instance (s16)255 * 127 + (s16)255 * 127 = -766 (= (s16)64770).

Intel MKL-DNN does protect users from this potential overflow. To avoid it either the data should be good or input should actually be u7, not u8. Our experience with Intel Caffe int8 inference showed that data distribution is typically good enough and an overflow doesn't happen (speculation: that's because mean(src_u8) << 127). That why we didn't do any changes to this sequence.
// BTW, with VPDPBUSD the problem will be gone [1].

To support a broader set of topologies we had to extend convolution with s8 (*) s8 -> s32 case. The problem here is that there is no such instructions in ISA. We had to emulate s8 * s8 via u8 * s8. For that instead of computing:

    dst_s32 <- src_s8 (*) wei_s8

we actual compute:

    dst_s32 <- (src_s8 + 128) (*) wei_s8 - 128 (*) wei_s8 = new_src_u8 (*) wei_s8 - compensation_s32

Note, that new_src_u8 (*) wei_s8 follows the same accumulation chain as mentioned above and has the same problem with a potential overflow. But it turned out that this overflow is no more just a potential but typically happens. That makes accuracy go down to almost zero. That's most likely because the data distribution for new_src_u8 is shifted away from 0 (mean(new_src_u8) = mean(src_s8) + 128 ~= 128).

To make the accuracy good on Skylake again we decided to shrink the weights by factor of two. So the real compute is:

    dst_s32 <- 2 * (s32) ((src_s8 + 128) (*) (wei_s8 / 2)) - 128 (*) wei_s8

That helped. Note that for HW that supports VPDPBUSD there is no need in this magic.

Phew...
So your HW is SandyBridge. Alas, we don't have an optimized int8 support for CPUs below Skylake server. The implementation would emulate s8 * s8 operations using double precision gemm (dgemm). That should be slower than even f32 convolution. But that should be good enough to make tests with int8. The sequence of operations repeat Skylake's ones. Whether this good or not -- I don't know (we've never thought about thoroughly). But we definitely need to document the behavior, to avoid the confusion.

Also we don't have plans to optimize int8 operations for

Thanks for rising this! And sorry for a long explanation.
Please let me know if you need further details or if the explanation is not clear.

---

[1]. About VNNI on Wiki-chip: https://en.wikichip.org/wiki/x86/avx512vnni
[2]. About INT8 inference on modern Intel HW: https://ai.intel.com/lowering-numerical-precision-increase-deep-learning-performance/

[SafeteeWoW]

Thank for your reply.
The behavior on int8 conv definitely should be documented.

I was planning to use mkl-dnn to accelerate our signed int8 convolution. But the acceleration code must be effectively the same as the naive implementation, so I cannot use mkl-dnn due to the precision lost in int8 convolution. I will write intrinsic to accelerate my implementation instead.

Last question, do you have any possible recommend way to optimize s8*s8->s32 on Skylake?

[emfomenk]

If your implementation cannot accept the potential overflow in int16 (as described above), then the only way to handle that would be:

    src_s16 <-- up-cast src_s8
    wei_s16 <-- up-cast wei_s8
    tmp_s32 <-- VPMADDWD(src_s16, wei_s16)
    dst_s32 <-- VPADDD(dst_s32, tmp_s32)

But in theory, this sequence would be a little slower than using vfmadd231ps in case of f32.

[emfomenk]

BTW, this sequence should also work for SandyBridge+ (with XMM registers / _m128i) and Haswell+ (with YMM registers / _m256i).

[SafeteeWoW]

Appreciate for your answer. I will try to implement this.

[chenqy4933]

I think this is a terrible design, anyway there should not precision problem of fix point computation, You should compute an other compensation_s32 when weight is not even. Then fix the result.