Rasa: tensorflow_embedding intent classifier accuracy

[therahulkumar]

Rasa NLU version: 0.12.3

Operating system (windows, osx, ...): osx

Content of model configuration file:

language: "en"

pipeline:
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"

Issue:
tensorflow_embedding intent classifiers accuracy decreases with increasing number of epochs, but Ideally it should overfit the data and its accuracy should increase by increasing the number of epochs. Is there anything wrong or what parameters should I tune to increase the train accuracy, below is the training log

epoch 10 / 300: loss 0.518545925617218, train accuracy : 0.961
epoch 20 / 300: loss 0.3817595839500427, train accuracy : 0.960
epoch 30 / 300: loss 0.3826253414154053, train accuracy : 0.957
epoch 40 / 300: loss 0.3831769824028015, train accuracy : 0.949
epoch 50 / 300: loss 0.35996705293655396, train accuracy : 0.937
epoch 60 / 300: loss 0.39998844265937805, train accuracy : 0.927
epoch 70 / 300: loss 0.4699452221393585, train accuracy : 0.934
epoch 80 / 300: loss 0.3826805055141449, train accuracy : 0.917
epoch 90 / 300: loss 0.34255778789520264, train accuracy : 0.920
epoch 100 / 300: loss 0.342288076877594, train accuracy : 0.933
epoch 110 / 300: loss 0.43107008934020996, train accuracy : 0.924
epoch 120 / 300: loss 0.3805781900882721, train accuracy : 0.915
epoch 130 / 300: loss 0.30855792760849, train accuracy : 0.923
epoch 140 / 300: loss 0.3602738380432129, train accuracy : 0.913
epoch 150 / 300: loss 0.33129388093948364, train accuracy : 0.918
epoch 160 / 300: loss 0.41288983821868896, train accuracy : 0.920
epoch 170 / 300: loss 0.30190521478652954, train accuracy : 0.925
epoch 180 / 300: loss 0.39234814047813416, train accuracy : 0.921
epoch 190 / 300: loss 0.3541390597820282, train accuracy : 0.925
epoch 200 / 300: loss 0.3109642267227173, train accuracy : 0.913
epoch 210 / 300: loss 0.3123457431793213, train accuracy : 0.908
epoch 220 / 300: loss 0.3863474726676941, train accuracy : 0.920
epoch 230 / 300: loss 0.42748862504959106, train accuracy : 0.914
epoch 240 / 300: loss 0.33208930492401123, train accuracy : 0.919
epoch 250 / 300: loss 0.3564454913139343, train accuracy : 0.912
epoch 260 / 300: loss 0.29999467730522156, train accuracy : 0.921
epoch 270 / 300: loss 0.3301870822906494, train accuracy : 0.925
epoch 280 / 300: loss 0.4273218512535095, train accuracy : 0.920
epoch 290 / 300: loss 0.3561229109764099, train accuracy : 0.919
epoch 300 / 300: loss 0.44117972254753113, train accuracy : 0.924

[Ghostvv]

@princerk could you please share your training data, so we could try to reproduce it

[therahulkumar]

@Ghostvv I can not share the data because of the proprietary reasons, but this works fine with small datasets but fails with large datasets. The above log is for dataset with 6000 examples and 90 intent classes. I also checked with datasets with 8500 examples with 130 intent classes, its performance decreased even further giving the train accuracy .77 and validation accuracy .72, I tried with changing the regularisation parameters and increasing the number of layers but that also did not help. I checked with spacy_sklearn intent classifier its performs better giving the train accuracy .99 and validation accuracy .80

[Ghostvv]

@princerk did you try to increase embedding dimension for larger datasets?

[therahulkumar]

@Ghostvv I did not try with changing the "embed_dim" value, Let me try I will update you with the result.

[therahulkumar]

Thanks @Ghostvv, Yes by changing the embed_dim I am able to get good train and validation accuracy. I changed embed_dim to 50, now I get .98 on train and .92 on validation set.