Mastodon: Language Filter Not Working

[lynkono]

Expected behaviour

The languages that I selected will be shown to me

Actual behaviour

Languages that I did not select are still shown to me.

Steps to reproduce the problem

Go to Settings > Preferences > Other > Language Filter and select the language you want toots to be shown. Go to a local timeline and see if the languages you selected are only there.

Specifications

Google Chrome on iOS

[Brawaru]

Thanks for submitting your first issue, @lynkono!

Unfortunately I was unable to reproduce it at mastodon.social.

This can happen because many languages are actually incorrectly detected as English (either the post is too short or letter set is too similar) and not much users set language that they post in in Settings. The filter itself is working, but you still see posts that are incorrectly labeled as English posts.

Related issues: #11118.

[lynkono]

That's strange. I just checked my local timeline and found a post in Thai, which uses a different alphabet than English. How did it detect it as English and if not, then why would the person put their account under English if that's the case?

[Brawaru]

Can you please link the post that was incorrectly detected?

[lynkono]

https://mas.to/@mizuboshi340/104276455190708313
This was in my local timeline

[Brawaru]

Thank you!

The problem with this post is clear: it does not meet the word count criteria (neither it has 4 of the words separated by spaces, nor it matches “reliable characters regular expression”), therefore language detection being skipped in favor of account's default posting locale, which is set to default value — English.

I actually believe this is an unintended bug, giving that “The Thai language does not use spaces between the words in a sentence” (source). The bug can be fixed by adding \p{Thai} to RELIABLE_CHARACTERS_RE.

Do you have any more examples besides Thai one?

[lynkono]

I remember seeing a lot of Spanish posts a few days ago, but I don't see any examples now. They also share the similar alphabet as English, so I don't know how that can be filtered out.

[Brawaru]

Hi! I've got some free time and can fix issue related to Thai language. Did you find any other languages not detected correctly?