Rasa: TrainingData.merge() doesn't check for duplicate examples per intent

[MetcalfeTom]

e.g. by merging

## intent:greet
- hi

with

## intent:greet
- hi

we get

## intent:greet
- hi
- hi

[akelad]

just when merging? do even check for duplicates within an intent?

[MetcalfeTom]

You're right, there's nothing implemented. The only data processing done for the intents upon building TrainingData() is in sanitize_examples() which removes whitespace

[akelad]

@MetcalfeTom is this still relevant?

[MetcalfeTom]

It is still relevant for features like interactive learning where we dump new NLU data. I think it functions well as a community issue

[hsm207]

I want to work on this

[akelad]

awesome! let us know if we can help with anything

[hsm207]

@akelad what would the unit test using @MetcalfeTom example look like? Should I create a markdown file containing the given examples or can each intent be represented as an instance of TrainingData?

[akelad]

@hsm207 you can take a look at the tests written here: https://github.com/RasaHQ/rasa/blob/master/tests/nlu/base/test_training_data.py

[hsm207]

@akelad I've made PR #4414 for your review.

[hsm207]

@akelad This issue can be closed now.