Rspamd: [BUG] R_MIXED_CHARSET false positive with czech language
Describe the bug
All e-mails with correctly encoded Czech diacritics in From, To, Subject, or body are incorrectly marked with R_MIXED_CHARSET symbol.
It started after updating rspamd from 1.9.4 to 2.0.
Steps to Reproduce
cat <<EOF | rspamc | grep R_MIXED_CHARSET
From: =?utf-8?q?Testovac=C3=AD_odes=C3=ADlatel?= <[email protected]>
To: =?utf-8?q?Testovac=C3=AD_p=C5=99=C3=ADjemce?= <[email protected]>
Subject: =?utf-8?q?P=C5=99i=C5=A1el_nov=C3=BD_mail!?=
Date: Mon, 18 Nov 2019 09:31:51 +0100
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_29989030.998171461718"
------=_NextPart_29989030.998171461718
Content-Type: text/plain;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Od: [email protected], P=C5=99edm=C4=9Bt: Od knihtisku k tisk=C3=A1r=
n=C4=9B.
------=_NextPart_29989030.998171461718--
EOF
Symbol: R_MIXED_CHARSET (8.33)[subject]
Versions
Rspamd daemon version 2.0
Linux emx1 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux
Additional Information
Probably releated to 4839327.
prehor
All 30 comments
Czech is particularly difficult for this rule. Do you have some more samples?
vstakhov
on 26 Nov 2019
Hi @vstakhov, here is an archive of czech FreeBSD mailing list: czech-language-eml.tar.gz with a lot of emails.
prehor
on 28 Nov 2019
+1 on this. Any quoted-printable (observed on UTF-8 and iso-8859-2 so far) seems to trigger this. It doesn't seem to happen with base64-encoded data.
An example I had for my report before I found out this one already exists (note that the subject is the same, only differently encoded):
cat <<EOF | rspamc | grep R_MIXED_CHARSET
Date: Thu, 03 Dec 2019 08:03:23 +0200
From: [email protected]
To: [email protected]
Subject: =?UTF-8?Q?Fwd=3A_z=C3=A1silka_slu=C5=BEby_=C3=9Aschovna=2Ecz?=
TextTextText
EOF
Symbol: R_MIXED_CHARSET (3.75)[subject]
cat <<EOF | rspamc | grep R_MIXED_CHARSET
Date: Thu, 03 Dec 2019 08:03:23 +0200
From: [email protected]
To: [email protected]
Subject: =?UTF-8?b?RndkOl96w6FzaWxrYV9zbHXFvmJ5X8Oac2Nob3ZuYS5jego=?=
TextTextText
EOF
<no output>
md5-4e29529727963bedb9d31b2951dcd749
Rspamd daemon version 2.2
Disassembler0
on 4 Dec 2019
Also, checking the mailing list linked above, there's a lot of emails without diacritics, so adding also a few of mine - all of them anonymized newsletters, so mostly spam anyway, but certainly not with mixed charset. - cz-mails.tar.gz
Disassembler0
on 4 Dec 2019
Hit the same issue with
Subject: =?utf-8?Q?Novinky=20kolem=20odes=C3=ADl=C3=A1n=C3=AD=20Original=20Prusa=20MINI?=
which is actually "Novinky kolem odesÃlánà Original Prusa MINI"
There seems to be something wrong with diacritics because in this subject there are letters used in other languages (not something fancy as _ř_ or _ů_ which is czech speciality)
analogic
on 20 Jan 2020
The same problem
=?utf-8?q?Vyj=C3=A1d=C5=99en=C3=AD_-_plyn?=
decoded "Vyjádřenà - plyn"
á,ř,à are correct czech letters
xmacan
on 28 Jan 2020
+1
sirdemoncze
on 30 Jan 2020
Version 2.3 did not fix the problem. All the examples given above are still marked with R_MIXED_CHARSET with considerable score.
Disassembler0
on 5 Feb 2020
No, not all.
Only those where czech language is not detected correctly. I see no ways how to detect language where a part of message is written in one language and another part is written in e.g. English.
Another important problem is how to distinguish that from real obfuscation attempts which are quite common in spam.
So the only thing I can suggest is to reduce score of this rule if you intend to receive messages that are in czech. I'm sorry but I see no other way to fix your language support without breaking other languages.
vstakhov
on 5 Feb 2020
Probably, we can improve language (mis)detection. I'd better try this way to be honest. E.g. czech language currently does not define stop words. If we can define some common words as stop words for czech language then it would help Rspamd to detect language properly and thus skip diacritics checks.
Do you have any ideas about that? You can check the existing stop words here: https://github.com/rspamd/rspamd/blob/master/contrib/languages-data/stop_words
Patches for Czech language are very wellcome!
vstakhov
on 5 Feb 2020
PR for Czech stop words opened as #3253.
I'm still wondering why does it matter if the string is quoted printable or base64 encoded (the difference in https://github.com/rspamd/rspamd/issues/3156#issuecomment-561526700). UTF-8 encoding is given in both and the string itself is also the same.
Disassembler0
on 5 Feb 2020
This bug also affects Slovak language. Curiously, only the From header affects the score?
Both From an To UTF-8 encoded:
# cat <<EOF | rspamc | grep R_MIXED_CHARSET
From: =?UTF-8?Q?Ve=C4=8Dern=C3=AD=C4=8Dek?= <[email protected]>
To: =?UTF-8?Q?Ve=C4=8Dern=C3=AD=C4=8Dek?= <[email protected]>
Date: Sat, 08 Feb 2020 10:28:00 +0200
Subject: Test
Test
EOF
Symbol: R_MIXED_CHARSET (7.50)[subject]
#
Only From UTF-8 encoded:
# cat <<EOF | rspamc | grep R_MIXED_CHARSET
From: =?UTF-8?Q?Ve=C4=8Dern=C3=AD=C4=8Dek?= <[email protected]>
To: Vecernicek <[email protected]>
Date: Sat, 08 Feb 2020 10:28:00 +0200
Subject: Test
Test
EOF
Symbol: R_MIXED_CHARSET (7.50)[subject]
#
Only To UTF-8 encoded:
# cat <<EOF | rspamc | grep R_MIXED_CHARSET
From: Vecernicek <[email protected]>
To: =?UTF-8?Q?Ve=C4=8Dern=C3=AD=C4=8Dek?= <[email protected]>
Date: Sat, 08 Feb 2020 10:28:00 +0200
Subject: Test
Test
EOF
#
lowstar
on 8 Feb 2020
I have installed ispconfig with rspamd on Debian 10 and I can see this issue happening with all emails containing slovak language content. Comparing czech and slovak in regard to unicode ranges it's all the same except of a few minor differences.
I imagine this might trigger for any european language with non ascii characters in their alphabed such as french, italian, portuguese, croatian, slovenian, polish and all other slavic languages.
I don't know about languages such as greek, serbian, russian, ukrainian with non latin based alphabet.
If this can't really be solved it would be nice to be able to override score of this check not to be +5 but way way lower because for example in my case 90% of email content is slovak and all of these emails have score +5 added to their spam score and that's really undesired.
How do I workaround this please?
pvagner
on 13 Feb 2020
How do I workaround this please?
On Ubuntu, I've reduced the score with following config in /etc/rspamd/local.d/headers_group.conf
symbols = {
"R_MIXED_CHARSET" {
weight = 1.0;
}
}
I guess it'll be the same on Debian.
Disassembler0
on 13 Feb 2020
Replying to @pvagner's question from #3253 here to keep the discussion consistent.
@Disassembler0 Shal I do something like this for slovak? What source have you used e.g. hunspell dictionary or something else?
I don't fully understand the matter but I'd be happy to make it work for me and other people writing emails in slovak.
I have used https://github.com/Alir3z4/stop-words as the base (these are the words used by python stop-words module), but then realized that stop words in rspamd aren't actually stop words in the correct sense (ie. words which are so common that they should be skipped in search, indexing etc.), but instead words which are common for the language. So I have checked also frequency lists and added also words used in email communication like greetings, farewells, attachments etc. Note that the words should be unique for that particular language, which may be a problem for Czech vs. Slovak. Štúr should have tried harder. :P
On the other hand, I find this approach unfortunate at best.
- There are crapload of languages and dialects around the world and this problem affects pretty much any language using latin with diacritics. There are whole projects and libraries which took decades to develop for analyzing written text and inferring the language from it, and most of them still fail, if you use weird enough dialect. (I want to see a mail in východňár Slovak! :) )
- Moreover, this feature can be circumvented really easily. Just sprinkle a few stop words from the word list (currently 10, according to https://github.com/rspamd/rspamd/commit/cd1deb3c19e84b759924c5208d2e1405be3b857a) and voila, you have a get outta ~jail~ R_MIXED_CHARSET free card and you can do whatever you want with mixing charsets anywhere else in the mail.
- Last but not least, multi-language mails are pretty common. In my previous company, almost any anouncement went out in English, Czech and French, so I wonder which language would be selected by rspamd.
Simply put, I think that the language guessing is a misfeature and the charset mixing should be detected on lower level. Unicode ranges or something like that.
Disassembler0
on 13 Feb 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 13 Apr 2020
Hello, the problem is still there.
hubiktomas
on 14 Apr 2020
I confirm that the problem is still there too. The problem will not disappear by ignoring the issue for a long enough time…
jirutka
on 14 Apr 2020
Ok, I will make it disappearing.
vstakhov
on 14 Apr 2020
You’re closing confirmed bug report before actually fixing the bug?! o.O
jirutka
on 14 Apr 2020
Yes, I'm inclined to close toxic threads. In general, if you have something to suggest then please send pull requests.
I'm aware of the issue with Czech and other slavian languages that use latin characters but I have no ideas about how to fix that. The same claim is valid for emails with multiple languages.
Hence, constant bothering me about this issue will not help at all. You are free to fix this bug by yourself of course.
Patches are always wellcome.
vstakhov
on 14 Apr 2020
OK. It seems that R_MIXED_CHARSET is unusable in our language environment and we need to turn it off. It would be nice if you mentioned this problem in the documentation.
It would be worth considering whether R_MIXED_CHARSET should have such a high score by default and possibly whether it is usable at all due to these problems.
prehor
on 14 Apr 2020
The site is open for modifications: https://github.com/rspamd/rspamd.com :)
andryyy
on 14 Apr 2020
You can easily suggest changes for the documentation in rspamd.com repo.
I'm not sure if reducing the weight of symbol that causes FPs for like less than 1% of users is a good thing. But I'm not very happy with the current language detection logic: there are many things to be improved TBH.
vstakhov
on 14 Apr 2020
What does R_MIXED_CHARSET actually do?
I see messages generated by thunderbird being scored up that have UTF-8 From: and plain Subject:, is this supposed to be the way it works?
$ cat mixedchar && cat mixedchar | rspamc
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: =?utf-8?Q?Franz_Georg_K=C3=B6hler?= fgkoehler@openunix.de
Mime-Version: 1.0 (1.0)
Subject: Test
Date: Thu, 13 Aug 2020 14:31:42 +0200
Message-Id: 85F283F1-CE2B-470A-8375-34778AAE3670@openunix.de
To: [email protected]
Das ist ein Test.
Results for file: stdin (0.079 seconds)
[Metric: default]
Action: no action
Spam: false
Score: 3.91 / 13.00
Symbol: ARC_NA (0.00)
Symbol: BAYES_HAM (-0.23)[72.54%]
Symbol: DMARC_NA (0.00)[openunix.de]
Symbol: FREEMAIL_ENVRCPT (0.00)[test.de]
Symbol: FREEMAIL_TO (0.00)[test.de]
Symbol: FROM_HAS_DN (0.00)
Symbol: HFILTER_HOSTNAME_UNKNOWN (2.50)
Symbol: MID_RHS_MATCH_FROM (0.00)
Symbol: MIME_GOOD (-0.10)[text/plain]
Symbol: MIME_TRACE (0.00)[0:+]
Symbol: MV_CASE (0.50)
Symbol: RCPT_COUNT_ONE (0.00)[1]
Symbol: RCVD_COUNT_ZERO (0.00)[0]
Symbol: R_DKIM_NA (0.00)
Symbol: R_MIXED_CHARSET (1.25)[subject]
Symbol: TO_DN_NONE (0.00)
Message-ID: [email protected]
fgkfgkfgk
on 14 Aug 2020
The result with Hungarian email is same:
Subject: =?UTF-8?Q?A_k=C3=B6zv=C3=A9c=C3=A9_k=C3=B6z=C3=A9rdek!?=
Symbol: R_MIXED_CHARSET (6.25)[subject]
dacrhu
on 24 Aug 2020
The result with Hungarian email is same:
Subject: =?UTF-8?Q?A_k=C3=B6zv=C3=A9c=C3=A9_k=C3=B6z=C3=A9rdek!?= Symbol: R_MIXED_CHARSET (6.25)[subject]
Try this one, I just did the same, adjust the score as you wish:
/etc/rspamd/local.d/headers_group.conf
symbols = {
"R_MIXED_CHARSET" {
weight = 1.0;
}
}
Alternative:
pongraczi
on 27 Aug 2020
The result with Hungarian email is same:
Subject: =?UTF-8?Q?A_k=C3=B6zv=C3=A9c=C3=A9_k=C3=B6z=C3=A9rdek!?= Symbol: R_MIXED_CHARSET (6.25)[subject]Try this one, I just did the same, adjust the score as you wish:
/etc/rspamd/local.d/headers_group.conf symbols = { "R_MIXED_CHARSET" { weight = 1.0; } }Alternative:
Thank you, working fine.
dacrhu
on 27 Aug 2020
How do I workaround this please?
On Ubuntu, I've reduced the score with following config in
/etc/rspamd/local.d/headers_group.confsymbols = { "R_MIXED_CHARSET" { weight = 1.0; } }I guess it'll be the same on Debian.
Thank you, this works very well on Arch too!
chrboesch
on 12 Nov 2020
Oh, so I'll have to disable that symbol and set it to 0. It's totally broken, same for very simple German text. It's certainly not the Czech, it's the check (pun not intended). This is not something for a real mail server.
Suggested fix: Remove that rule. Very easy.
PS: It's not this thread that's toxic, it's the maintainer's behaviour. Think about it.
ygoe
on 23 Nov 2020
Related issues
stephdl
·
6Comments
andris9
·
5Comments
dothebart
·
3Comments
mpizzoli
·
3Comments
mguesdon
·
5Comments
Most helpful comment
Hello, the problem is still there.