Parlai: How can I get started with using the dialogue safety model?

[josharnoldjosh]

How can I get started with using the dialogue safety model?

I saw the script for downloading the model and the link to the paper here: https://github.com/facebookresearch/ParlAI/blob/master/projects/dialogue_safety/README.md

But I am not sure how to use this model? My goal was but be able to create an instance of the model, something like so:

model = Model()

And then to be able to classify text as either safe or unsafe (using a pretrained model without training myself) like so:

text_is_safe = model.classify(text)

Does parl.ai support this functionality? Where would I get started with this?

I saw a worlds.py for the blender model I believe: https://github.com/facebookresearch/ParlAI/blob/master/parlai/chat_service/tasks/chatbot/worlds.py

Would I create a run.py and a worlds.py that uses the dialogue safety classifier model?

Thank you for your time! I apologize if this is a trivial question

[jxmsML]

Hi @josharnoldjosh,

You can load the pretrained single-turn safety model from this file https://github.com/facebookresearch/ParlAI/blob/master/parlai/utils/safety.py

Simply run the following:

from parlai.utils.safety import OffensiveStringMatcher, OffensiveLanguageClassifier
offensive_classifier = OffensiveLanguageClassifier()

which creates the model and then,

text_is_unsafe = text in offensive_classifier

where text_is_unsafe is True if the text is UNSAFE.

If you'd like to tune your own threshold, you can instead run:

text_is_unsafe, proba = offensive_classifier.contains_offensive_language(text)

where proba is the probability of the predicted class.

[josharnoldjosh]

Thank you!