Lila: Suggestion: Auto-flag offensive comment for review

[ddugovic]

Perhaps forum posts and/or Q&A questions with a severely negative sentiment analysis could be flagged for moderator review?
https://cloud.google.com/blog/products/gcp/new-features-in-the-google-cloud-natural-language-api-thanks-to-your-feedback

[Assios]

Relevant: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge

[sophiawisdom]

@ddugovic How many posts are we getting per month on the forums? Google's sentiment API is free for less than 5,000 posts per month. The Q&A fits well under that - I looked 15 days back and there were 80 posts, with an average of less than two answers per question, so that's like 400 posts per month. I'm not really sure how to measure the forums in the same way.

[ddugovic]

Let's see... 5,000 posts a month would be about 150-170 posts per day. I don't frequent all the forums enough to know whether we hit that or not.

[sophiawisdom]

@ddugovic Beyond that point we need money. How does the site currently run? If it runs on e.g. Google Apps then it wouldn't be too hard to start something like this. Typically Google is going to want a credit card to run these APIs. The cost per month under any circumstance is likely to be low. As before, free for the first 5k/month but the cost after that is only $1/1,000 posts.

[TitoYak211]

IDK, but bear in mind that Lichess has a very strong open-source, free, and ad-free outlook. Google by definition does not fit this sort of atmosphere.

[mmarkell]

We could just have a blacklist of bad words and check if the comment contains one, rather than doing fancy sentiment analysis, at least for a start.

[thomas-daniels]

A blacklist is what we currently have, and use in our communications auto-reporting system. But a blacklist has no sense of context -- a "bad word" in itself does not imply an offensive comment/message.

[sophiawisdom]

@ProgramFOX If all we're doing is flagging, I don't think the lack of context is a huge concern - the ultimate decision is on the moderators, and it's their responsibility to look at context. If we want to improve the sophistication of the flagging, we could still follow the sentiment analysis path and just use someone else's model and run it locally.

[thomas-daniels]

A blacklist cannot catch everything. There are always false positives with it so you can only add the words to it that have a reasonable accuracy and will not make you drown in false positives. A blacklist won't just catch messages like "I wish you die" or "your chess is a joke and you are worth nothing" and variations - sentiment analysis could.