Onednn: Add 'vzeroupper' at the end of primitives?

[CaoZhongZ]

We encountered several AVX/SSE transitional penalty when switch from MKL-DNN to NumPy (called libm). The SSE performance is 3x worse when upper bits of registers is non-zero. We are covering these operations with new implementations but it requires time. We can also add OpenMP regions to execute the instruction after called MKL-DNN. But it seems much cheaper to just add it to the end of OpenMP region of every primitives, left SIMD registers a clean state when leaving MKL-DNN. Is that feasible?

[rsdubtso]

Can you please check if this patch fixes your problem?

[vamsi-sripathi]

If I'm reading the patch correctly, vzeroupper would be generated for all Intel architectures that support AVX ISA. But, vzeroupper is not needed on Xeon Phi (KNL) and adds significant overhead. Related discussion here, https://software.intel.com/en-us/forums/intel-isa-extensions/topic/704023
We could protect vzeroupper generation on KNL by modifying the conditional from:
if(mayiuse(avx)) -> if(!mayiuse(avx512_mic) && mayiuse(avx))

[rsdubtso]

Hi Vamsi, you are right, the KNL should be excluded.

[CaoZhongZ]

Thank you! I'll try this patch and come back here later.

[CaoZhongZ]

Ah, one case left after the patch. It might caused by MKL gemm path or reorder. Do MKL take care of AVX registers?

[rsdubtso]

I've updated the gist. The (closed-source) MKL should be fine. @4pao, can you please try again?

[CaoZhongZ]

I'll try it. wait for a while.

[CaoZhongZ]

The patch is necessary. Further more, we did a investigation found out that large number of slow down cases resulted from "C/C++" parts of MKL-DNN. Because when we compiled MKL-DNN with '-march=native', the performance of SSE is down. If we don't, no performance loss. So I might be wrong about adding vzeroupper might solve this problem, it scattered everywhere if compiled with certain flags.

[rsdubtso]

All the compilers should be adding vzeroupper on transitions. I would suggest trying a newer gcc (guessing from the -march=native). I will still push this patch to the master, but will close this issue as resolved. Feel free to reopen it if necessary.