Rasa: Defining generic entities for a single intent

[anantha-madhava]

rasa NLU version (e.g. 0.7.3): 0.8.0-a10

Used backend / pipeline (mitie, spacy_sklearn, ...): spacy_sklearn

Operating system (windows, osx, ...): Ubuntu 16.04

Issue:
I have been trying to use rasa_nlu for triggering builds on Jenkins server. I have a good amount of training data with many Jenkins jobnames as entities and the intent is 'build_job'.
My Jenkins server has more than 100 jobs. Is there a way I can make the entity very generic so that I do not have to have all 100 job names as entities in my training data?

Here's my sample json file:

{
"rasa_nlu_data": {
"entity_examples": [],
"intent_examples": [],
"common_examples": [
{
"text": "please build JobName1",
"intent": "build_job",
"entities": [
{
"start": 13,
"end": 20,
"value": "JobName1",
"entity": "job_name"
}
]
},
{
"text": "could you please build JobName2 for me?",
"intent": "build_job",
"entities": [
{
"start": 13,
"end": 20,
"value": "JobName2",
"entity": "job_name"
}
]
}
]
}
}

[tmbo]

@anantha-madhava thanks for your post. I get the issue, I am not sure though how to approach it yet. Essentially you want to way of saying "here is one training sample: 'please build JobName1'" please automatically replace the labeled entity JobName1 with the following entities ['JobName2', 'JobName3'] during training and add the generated training samples to the training set. Does that make sense?

[amn41]

In general rasa NLU should be able to generalise to unseen words (job names) - so you shouldn't need all of them as training data.

I think it's worth trying the CRF entity module

[anantha-madhava]

Hello Team,
My expectation was as mentioned by @amn41 . Just to elaborate a bit more -
Say, I have given 10 questions in my training data, with project_spacy, build_mitie, mitie_nightly_build... [10 such different job names in Jenkins] as entities for the same intent of 'build_job'. I expect rasa_nlu to understand when a user calls for a build of rasa_nightly_build or any JobName'X' which is not explicitly mentioned in my training data.

This has not been happening with rasa_nlu V0.8.0-a10. How can we achieve this?

Also please let me know what is CRF entity module and how do I make use of it?

[xiaodong-zhu]

@tmbo
Hello, trying to migrate my app from api.ai.
in api.ai, entities have to be defined first, like the entity 'fruit' has elements 'apple', 'orange' and 'banana', and then when we provide training samples entities can be referenced. But looks like in rasa we got to specifically provide three samples containing 'apple', 'orange' and 'banana' respectively. I'm wondering is there a way to define entities first which can then be used in training samples.
And one more question about entity as well. api.ai has such an option called "Allow automated expansion" in entity definition, elements not specifically listed in an entity are regarded invalid if this option is switched off. But in rasa, even if only 'apple', 'orange' and 'banana' are provided, 'cherry' would also be filled as an entity value in some cases. Can I disable this?
btw, thank you very much for this amazing tool. our app servers are located in Asia and api.ai is slow to respond and especially unstable. rasa is fantastic.

[imraansarwar]

@xiaodong-zhu Suggest you to have a look at "entity_examples" as specified here https://rasa-nlu.readthedocs.io/en/stable/dataformat.html
I'm not sure if that would work for your use case, do share your experience on that.

Thanks,
Imran

[tmbo]

Given "enough" training samples the entity recognition will generalize to new and unseen entities (based on surrounding words / context / POS taggs / ...). The emphasis is on "enough" and means around 5000 training samples for traditional NER systems, the CRF component can work with less though.

@xiaodong-zhu You don't need to specify entities or their values first, include them in the training samples and you should be fine. There is no way to disable "automated expansion". I'd suggest you check the entity value in your application code and go from there.

[xiaodong-zhu]

Hi @imraansarwar @tmbo,
thank you guys for your replies.
the data format does solve the problem of entity training, but personally, I'd go for a separate entity definition and bindings between entities and intents, as it brings decoupled data layers and better reusability, especially for generic entities like locations or names.
Currently I'm using a simple script to generate the annotated sample files out of separate entity and intent data. I guess such tools could be plugged into rasa-nlu-trainer, where core rasa logics don't have to be changed while another way of training data can be provided.

Thanks

[anantha-madhava]

hello @tmbo , Thanks for that hint.
I managed to get "enough" samples of 29000 for the same build_job intent :) This sample contains all the possible Jenkins JobNames from one Jenkins server. The same json file also contains few sample 'greet' intents as well. With this, the training was successful and the new job names [which are not in the sample data] are recognized as correct entities.

However, when I ask an irrelevant question, the response was that of a build_job..!! Which is wrong and unexpected. Below is the screenshot of such a question where the intent and entity identification has gone wrong.

[tmbo]

@xiaodong-zhu I agree with you on that. feel free to propose a new format & as a bonus implement the parsing ;)

[tmbo]

@anantha-madhava Thats most likely because you only have the two intents greet and buildjob. The models will always select one of them. Hence, even if you pass in unrelated sentences, they will still be classified with one of your intents.

[thefuryelement]

can anybody tell me .. why am i not able to generate multiple entities here ?? i mean i have more than one entities in a statement that i have trained my model on.. but it would only show one or none