Onednn: When the GRU input is entered as 0, the result is different from the GRU of pytorch.

[kjyeon1676]

Hello, I am making DS2 (DeepSpeech2 model) model using GRU primitive. However, if you enter zero input into GRU primitive, the result is not the same as pytorch GRU. By the way, if enter 1, the result is the same as pytorch GRU. All network parameters were initialized to 1 and tested. Below is the code I wrote.

~c++
vector inputBuf(2 * 1 * 6, 0.0f) // (T, N, C) init value is zero
vector inputBuf2(2 * 1 * 6, 1.0f) // init value is one.
auto inputMem = memory({{2, 1, 6}, data_type::f32, memory_format::tnc}, engine, inputBuf.data());
... (weightslayer, weightsiter, bias setting using ldigo, ldgo)
... (direction : bidrectional_sum, kind: prop_forward_training)
auto gru_primitive = gru_forward::primitive(..., engine)
... (src, srciter, weightslayer, weightsiter reordering)
...
gru.execute(...) ...
~

DNNL GRU execute result using one input:

pytorch GRU execute result using one input:

DNNL GRU execute result using zero input:

pytorch GRU execute result using zero input:

@rsdubtso: edited for formatting

[zixuanweeei]

There are two versions of GRU supported by DNNL, LBR-GRU and GRU. From the document of PyTorch, it should use LBR-GRU primitive.
Pytorch: https://pytorch.org/docs/stable/nn.html#gru
DNNL: https://intel.github.io/mkl-dnn/dev_guide_rnn.html
Could you give a try? It would be highly appreciated.

[kjyeon1676]

Thank you :) I did't check the LBR-GRU T-T..
can i ask a one more question?..
from the document of DNNL(LBR-GRU), there is follow description.
-> For the bias tensor, we implicitly require the order of the gates to be u, r, o, and u'.
So, When I initialize bias, can i initialize as follows?

vector<float> biasBuf(numLayers * numDirection * 4(adding extra_bias) * hidden_state_size, 1.0f);
int offset = numLayers * numDirection * hidden_state_size;
for (int i = offset * 3; i < biasBuf.size(); i++) { // b_u' bias init.
    biasBuf[i] = 0.5f;
}
/* So, internal vector value as follow.
  1 1 1 1 1 1 1 1  --> b_u
  1 1 1 1 1 1 1 1  --> b_r
  1 1 1 1 1 1 1 1  --> b_0
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5  --> b_u'

[zixuanweeei]

@kjyeon1676 Yes, you are correct.

[kjyeon1676]

Thank you for your kindness !

[kjyeon1676]

I'm sorry to reopen it. T-T
thanks to zixuanweeei help, we're continuing to build the DS2 network.
I'm creating a network comparing the results with pytorch.

Because of reopen, bidirectional_sum result seems to be strange.
When i use to option about unidirectional_left2right and right2left, result same to pytorch.
However, When i use to option about bidirectional_sum, result is different.
Below is the code i Wrote.

// input shape and size.
// T(seqence_length) : 2, batch : 2, numLayer : 1, direction : 2, C : 6, H : 8
// input value : 0.0f, hidden state memory value : 0.0f;
// C : input channel, H : hidden_state_size
vector<float> inputBuf(T * batch * C, 0.0f);
auto hiddenMem = memory();

vector<float> userWeightsLayerBuf(numLayer * numDirection * C * 3 * H, 1.0f);
vector<float> userWeightsIterBuf(numLayer * numDirection * H * 3 * H, 1.0f);
vector<float> userBiasBuf(numLayer * numDirection * 4 * H, 2.0f);   // add extra_bias + 1
// The reason why 2.0 is given as the initial value is that pytorch GRU adds bias twice.
    int offset = numLayers * numDirection * H;
    for (int i = offset * 2; i < bias.size(); i++) {
        userBiasBuf[i] = 1.0f;
    }

 memory::dims weightLayerDims{numLayers, numDirection, C, 3, H};
 memory::dims weightIterDims{numLayers, numDirection, H, 3, H};

 // dimensional sequence : l d g o
 // num_layers, num_directions, num_gates(lbr_gru is extra_bias + 1), output_channels(H).
 memory::dims biasDims{numLayers, numDirection, 4, H};

 // create memory descriptor for user data.
 auto userWeightsLayerDesc = memory::desc(weightLayerDims, dataType::f32, memoryFmt::ldigo);
 auto userWeightsIterDesc = memory::desc(weightIterDims, dataType::f32, memoryFmt::ldigo);
 auto userBiasDesc = memory::desc(biasDims, dataType::f32, memoryFmt::ldgo);

 auto userWeightsLayerMem = memory(userWeightsLayerDesc, eng);
 auto userWeightsIterMem = memory(userWeightsIterDesc, eng);
 auto userBiasMem = memory(userBiasDesc, eng);

 write_to_dnnl_memory(userWeightsLayerBuf.data(), userWeightsLayerMem);
 write_to_dnnl_memory(userWeightsIterBuf.data(), userWeightsIterMem);
 write_to_dnnl_memory(userBiasBuf.data(), userBiasMem);

 // create memory descriptor for GRU data w/ no specified.
 auto weightsLayerDesc = memory::desc(weightLayerDims, dataType::f32, memoryFmt::any);
 auto weightsIterDesc = memory::desc(weightIterDims, dataType::f32, memoryFmt::any);
 auto biasDesc = memory::desc(biasDims, dataType::f32, memoryFmt::any);
 ...
 ...(omission)
 auto srcLayerDesc = memory::desc({srcLayerDims}, dataType::f32, memoryFmt::tnc);
 auto dstLayerDesc = memory::desc({dstLayerDims}, dataType::f32, memoryFmt::tnc);
 auto dstIterDesc = memory::desc({dstIterDims}, dataType::f32, memoryFmt::ldnc);

 prop_kind kind;
 if (isTraining) {
        kind = prop_kind::forward_training;
 } else {
        kind = prop_kind::forward_inference;
 }

 auto gruFwdDesc = lbr_gru_forward::desc(kind,       // propagation kind
                                        direction,                            // direction
                                        srcLayerDesc,                     // srcLayerDesc
                                        memory::desc(),                 // srcIterDesc
                                        weightsLayerDesc,             // weightLayerDesc
                                        weightsIterDesc,               // weightIterDesc
                                        biasDesc,                          // biasDesc
                                        dstLayerDesc,                    // dstLayerDesc
                                        dstIterDesc);                      // dstIterDesc, flag

 auto gruPd = lbr_gru_forward::primitive_desc(gruFwdDesc, eng);
 if (isTraining) {
      data.workspaceMem = memory(gruPd.workspace_desc(), eng);
 }
 // input == memory object
 auto srcLayerMem = input;
 // omission.. (reordering and lbr_gru_forward execute)
 ...

and below picture is execution result

DNNL left2right result

DNNL right2left result

pytorch result - unidirection

In my guess, bidirectional_sum result should be a result of adding 1 and 2 rows.
Therefore, DNNL results should be the same as pytorch. However, the results were different.

pytorch bidirectional result

DNNL bidirectional sum result

Please let me know if I miscoded or if the usage is different.
There is no example of GRU, so I do not know if it is right to do this. T-T

[mgouicem]

Thank you :) I did't check the LBR-GRU T-T..
can i ask a one more question?..
from the document of DNNL(LBR-GRU), there is follow description.
-> For the bias tensor, we implicitly require the order of the gates to be u, r, o, and u'.
So, When I initialize bias, can i initialize as follows?

vector<float> biasBuf(numLayers * numDirection * 4(adding extra_bias) * hidden_state_size, 1.0f);
int offset = numLayers * numDirection * hidden_state_size;
for (int i = offset * 3; i < biasBuf.size(); i++) { // b_u' bias init.
    biasBuf[i] = 0.5f;
}
/* So, internal vector value as follow.
  1 1 1 1 1 1 1 1  --> b_u
  1 1 1 1 1 1 1 1  --> b_r
  1 1 1 1 1 1 1 1  --> b_0
  0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5  --> b_u'

Hi @kjyeon1676 , the above code would be correct if the bias layout was gldo. However, we use ldgo for the bias layout, and the above code is correct only for numLayers = numDirection = 1 in this context.
To correctly produce what you want, you have to do the following:

for(int l = 0; l < numLayers; l++)
    for(int d = 0; d < numDirection; d++)
        for(int o = 0; o < hidden_state_size; o++)
            biasBuff[ o + hidden_state_size * (3 +  numBiasGates * ( d + numDirection * l))] = 0.5f;

[kjyeon1676]

@mgouicem I checked that the result with pytorch match. :)!!!!
Thank you for your kindness!!! :)