Rasa: Does rasa tensorflow now support GPU training?

[tushar1328]

Can Training of RASA be done on GPU? If yes please let me knw. @akelad @wrathagom

[wrathagom]

@Ghostvv is the expert here. I would imagine if you have installed the gpu enabled tensorflow though that yes it would take advantage of your GPU.

[JoeyFaulkner]

@wrathagom is correct! It should just automatically use your GPU if you follow https://www.tensorflow.org/install/ with GPU support. If you have any issues with rasa nlu/core after
following this installation then let us know as that would be useful information, but it should work just fine.

[tushar1328]

@wrathagom @JoeyFaulkner will sure let you knw if there is any issue for the same. will try in 2 days and let you knw. Thank you @wrathagom @JoeyFaulkner

[saxh]

@JoeyFaulkner @wrathagom I am sure of using tensorflow-GPU=1.11.0. But it's not used when training, can you help me?Thanks a lot.

[dvigneshwer]

For some reason, the rasa-nlu model training command is not utilizing my GPU memory (using a Tesla V-100 in an aws instance) - Is there any argument that I need to pass in specific for the code to use the GPU resources? I tried running the training command with just tensorflow-gpu package but that didn't work as well.

[kdesai2]

Hey I'm also facing this same issue, didn't find a solution anywhere on forum too

[akelad]

@Ghostvv any ideas?

[Ghostvv]

do other tf models use GPU on your computer?

[saxh]

font{
    line-height: 1.6;
}
ul,ol{
    padding-left: 20px;
    list-style-position: inside;
}



Yes.I reinstalled tensorflow-gpu and it working on rasa model






















                            ilshine




                                [email protected]








    签名由
    网易邮箱大师
    定制




On 12/12/2018 19:09，Vladimir Vlasov<[email protected]> wrote：

do other tf models use GPU on your computer?

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

[dvigneshwer]

@Ghostvv - yes other tf models are able to utilize the GPU resources

@saxh - Reinstalling tensorflow-gpu did not work for me

Training 55K strings take around 11 hours to complete using CPU resources.

[dvigneshwer]

@Ghostvv - It will be great if you can suggest any debugging techniques.

[Ghostvv]

@dvigneshwer can you check whether you use the same python environment for rasa, where you have reinstalled tensorflow-gpu

[saxh]

font{
    line-height: 1.6;
}
ul,ol{
    padding-left: 20px;
    list-style-position: inside;
}



Yes , it is. Or you can create a new environment for rasa and install tensorflow_gpu rather than tensorflow.

















                            ilshine




                                [email protected]








    签名由
    网易邮箱大师
    定制




On 01/2/2019 17:43，Vladimir Vlasov<[email protected]> wrote：

@dvigneshwer can you check whether you use the same python environment for rasa, where you have reinstalled tensorflow-gpu

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

[dvigneshwer]

@Ghostvv Yes it is the same environment where both rasa_nlu and tensorflow-gpu exist

@saxh - Created a separate conda environment for rasa training where I had installed rasa_nlu, tensorflow-gpu and other required packages - even that didn't consume the GPU resources while running training command

[dvigneshwer]

@Ghostvv Is it possible to integrate custom models into Rasa architecture? I trained a NER and intent recognition models using keras and would like to integrate it with Rasa.

[Ghostvv]

@dvigneshwer about the problem with tf GPU, unfortunately I don't know how to debug it.

For custom models you can create custom components and submit a PR for review

[veliander]

I also can't get it to train fast on a P3.2 AWS instance. It seems to detect the GPU (I'm running inside an Nvidia container nvcr.io/nvidia/tensorflow:18.02-py3 with pre-installed tensorflow-gpu), but then doesn't use it: the training speed is comparable to that on my iMac. My pipeline is

• name: "intent_featurizer_count_vectors"

- name: "intent_classifier_tensorflow_embedding"

2019-01-12 04:51:41 INFO rasa_nlu.model - Starting to train component intent_featurizer_count_vectors
2019-01-12 04:51:41 INFO rasa_nlu.model - Finished training component.
2019-01-12 04:51:41 INFO rasa_nlu.model - Starting to train component intent_classifier_tensorflow_embedding
2019-01-12 04:51:43.672399: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-12 04:51:43.672823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
totalMemory: 15.75GiB freeMemory: 15.34GiB
2019-01-12 04:51:43.672853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-01-12 04:51:44 INFO rasa_nlu.classifiers.embedding_intent_classifier - Accuracy is updated every 10 epochs
Epochs: 100%|██████████████████████████████████████████████████████████████████████████| 300/300 [24:00<00:00, 4.60s/it, loss=0.281, acc=0.949]
2019-01-12 05:15:45 INFO rasa_nlu.classifiers.embedding_intent_classifier - Finished training embedding policy, loss=0.281, train accuracy=0.949

[Ghostvv]

@veliander from the logs it seems that it uses the gpu

[Ghostvv]

@veliander about the performance gain, please see this discussion: https://github.com/RasaHQ/rasa_nlu/issues/1398