Onednn: Low performance of LSTM training forward and backward with some input values and weights

[benjamim93]

There seems to be some performance decrease in the training mode forward and backward of LSTM with some input values and weights. I edited the example code rnn_training_f32.cpp to make it a simple 1-layer LSTM with all weights (layer and iter) set to -0.1 and all input values set to 0.1. I ran the forward and backward for 100 times. The performance of forward and backward looks good. (time consumption of forward_training ~= 27, backward ~= 60) MKLDNN verbose log is here:
dnnl_lstm_normal_perf.log

But if I change all input values to -0.1 and left all other values unchanged, there is some huge performance decrease. (time consumption of forward_training ~= 500, backward ~= 1600) Log is here:
dnnl_lstm_low_perf.log

I noticed this problem when I was trying to boost the training process of LSTM using MKLDNN. The input values and weights were close to those in the tests above (in the range -0.2 = +0.2). I simply replaced the content of rnn_training_f32.cpp with my code, built and ran with

${MKLDNN_HOME}/build/examples$ ./rnn-training-f32-cpp

Code is appended. For the good performance one, please change the input values to 0.1 like:

std::vector<float> net_src(tz_volume(src_layer_dims), 0.1f);

Bad performance code.

(Edited by @rsdubtso for brevity -- moved the code to a gist.)

[rsdubtso]

I can reproduce it. Tagging @mgouicem for help.

[mgouicem]

Hi @benjamim93,
The performance gap is due to an underflow and computation with denormal values.
For each lstm cell, the GEMM computation will compute the following accumulation: (weights_layer_val * src_layer_val + weights_iter_val * src_iter_val)* hidden_size.
In the first iteration, src_layer_val = -0.1 and src_iter_val = 0, so the accumulation gives ~ 6.5. When computing sigmoid and tanh over this, you will have values close to 1.0, and the output hidden state will have values of ~ 0.76.
In the second iteration, the accumulation will give ~ -43. When computing sigmoid over this value, you will get very small values that are close to 0.0 but not zero (~ 2e-19). and when computing the output state of this second iteration, you will get a denormal value (namely ~ -1.9e-40). And these denormals will carry forward to the next iteration in the GEMM, and slow it down considerably.
For the following iterations, you will have denormals in the weight_iter * src_iter computation every other iteration.

If in a real workload you are affected by this slowdown, a good advice would be to flush the denormals to zero. You can refer to the above article on how to disable denormals depending on the compiler you use. If you want to do it in your code, you can use the following:

#include <xmmintrin.h>
//DAZ
_mm_setcsr( _mm_getcsr() | 0x0040 );
//FTZ
_mm_setcsr( _mm_getcsr() | 0x8000 ); 

I tried with the above code in the main function of the example you provided, and timings are about the same, independently of the input data.

[benjamim93]

@mgouicem Thank you for your explanation and advice. I added these lines of code and things worked well. Impressive.

I have two more questions:

1. Would DAZ and FTZ cause an obvious loss in accuracies? I tried on small sized inputs and weights no obvious loss appeared though.
2. Will operations be added to MKLDNN code to prevent this issue?

[emfomenk]

Hi @benjamim93,

1. I don't have very deep knowledge on different training algorithm and their sensitiveness to disabling denormals, but given that most of them work even with lower accuracy such as fp16 or bfloat16 I highly doubt disabling denormals would bring any accuracy issues.
2. Setting DAZ and FTZ flags should not be controlled by a library, but the high level application. From that perspective DNNL (Intel MKL-DNN) respects the MXCSR register. So I don't think we will add this functionality to the library.

[mgouicem]

Hi @benjamim93 ,
I will back @emfomenk on 1. Actually Tensorflow (here for GPU and using this class for CPU) set the ftz flag by default.

I will close this issue since it seems resolved. If you have any other questions or concern, feel free to reopen it or open a new one.