Rspamd: 'rspamc learn_spam' fails with 'HTTP error : 500, Unknown statistics error'

[yasuhirokimura]

Classification (Please choose one option):

• [ ] Crash/Hang/Data loss
• [ ] WebUI/Usability
• [x] Serious bug
• [ ] Ordinary bug
• [ ] Feature
• [ ] Enhancement

Reproducibility (Please choose one option):

• [X] Always
• [ ] Sometimes
• [ ] Rarely
• [ ] Unable
• [ ] I didn’t try
• [ ] Not applicable

Rspamd version:

1.8.3

Operation system, CPU:

FreeBSD 11.2-RELEASE amd64

Description (Please provide a descriptive summary of the issue):

About 2 weeks ago I found rspamc learn_spam fails as following.

yasu@eastasia[2041]% rspamc learn_spam ~/tmp/spam01.txt ~
Results for file: /home/yasu/tmp/spam01.txt (0.024 seconds)
HTTP error: 500, Unknown statistics error

I can't say exactly when but in the past it certainly worked well.

I removed all files under ${DBDIR} and start spam/ham learning from
scratch. At first 'rspamc learn_spam' completed successfully. But
after a week or so same error happened again.

rspamc learn_ham always succeeds.

Relevant logs (see details here):

When executing rspamc learn_spam following messages are written to
log file.

2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_message_parse: loaded message; id: <[email protected]>; queue-id: <undef>; size: 4210; checksum: <21b55fd227635acf41c417c420722c42>                          
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_mime_text_part_utf8_convert: converted from ISO_8859-1:1987 to UTF-8 inlen: 811, outlen: 811
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_mime_part_detect_language: detected part language: en       
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_task_process: learn error: Unknown statistics error         
2018-12-01 19:54:07 #752(controller) <731c60>; csession; rspamd_controller_learn_fin_task: cannot learn <[email protected]>: Unknown statistics error

Expected results:

rspamc learn_spam completes successfully.

Actual results:

rspamc learn_spam fails as following.

yasu@eastasia[2041]% rspamc learn_spam ~/tmp/spam01.txt ~
Results for file: /home/yasu/tmp/spam01.txt (0.024 seconds)
HTTP error: 500, Unknown statistics error

Configuration (e.g. rspamadm configdump module):

I don't make any customization about spam learning.

[vstakhov]

IDK: enable bayes debugging at least.

[yasuhirokimura]

Last month I made clean install of FreeBSD 12.0-RELEASE on my mail server and also made clean install of Rspamd 1.8.3 with debug_modules = ["bayes"]; in ${CONFDIR}/local.d/logging.inc. Then same problem happened last week. There is part of rspamd.log in following URL.

https://www.utahime.org/Rspamd/rspamd.20190112042927.log

If you need learned mail then I can provide it. And since the problem is reproducible now I can provide other cases if necessary. Does this help you investigating this issue?

[vstakhov]

Are you using sqlite backend?

[yasuhirokimura]

Yes, I think so.

yasu@eastasia[4693]% rspamadm configdump classifier.bayes.backend
*** Section classifier.bayes.backend ***
"sqlite3"
*** End of section classifier.bayes.backend ***
yasu@eastasia[4694]%

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sem-hub]

The same problem on FreeBSD with the log:
2019-05-28 13:57:17 #42580(controller) <9c769f>; csession; rspamd_task_process: learn error: Unknown statistics error, found on stage learn; classifier: (null)
2019-05-28 13:57:17 #42580(controller) <9c769f>; csession; rspamd_controller_learn_fin_task: cannot learn : Unknown statistics error, found on stage learn; classifier: (null)

The same backend: sqlite3

[Twilek-de]

I am seeing this problem now as well (I don´t know how long it has existed already. I had a redis backend. I have switched to sqlite and now it seems to work again.

[bong0]

Since a few weeks I'm having that issue as well, the log seems not so say more than the client output does. (the unknown statistics error)

My backed is redis, default install on Debian using the rspamd.com repo:

apt show rspamd
Package: rspamd
Version: 2.0-1~jessie
Maintainer: Vsevolod Stakhov <[email protected]>
[...]
APT-Sources: http://rspamd.com/apt-stable/ jessie/main amd64 Packages

Redis is not showing me errors in /var/log/redis.log;

Debug log:

[more ... "token" messages]
2019-10-27 15:28:06 #9075(controller) <966b1d>; bayes; bayes_learn_spam: token 10245623571907066200 <#rcv:45.155.37.0:esmtp:#F:Werner Klemm>: window: 2, total_count: 0, spam_count: 0, ham_count: 0
2019-10-27 15:28:06 #9075(controller) <966b1d>; bayes; bayes_learn_spam: token 539890791086949597 <#rcv:45.155.37.0:esmtp:#h:C-T:text/html; charset="UTF-8">: window: 3, total_count: 0, spam_count: 0, ham_count: 0
2019-10-27 15:28:06 #9075(controller) <966b1d>; bayes; bayes_learn_spam: token 1730648915295258020 <#rcv:45.155.37.0:esmtp:#hh:54e5275>: window: 4, total_count: 0, spam_count: 0, ham_count: 0
2019-10-27 15:28:06 #9075(controller) <966b1d>; csession; rspamd_task_process: learn error: Unknown statistics error, found when storing data on backend; classifier: (null)
2019-10-27 15:28:06 #9075(controller) <966b1d>; csession; rspamd_controller_learn_fin_task: cannot learn <[email protected]>: Unknown statistics error, found when storing data on backend; classifier: (null)

The error occurs no matter whether using rspamc or the web interface paste

[nahuelange]

I have the same error

[snowirbis]

Same here. FreeBSD 11.3 amd64 rspamd 2.2

[arnowelzel]

Same here, Ubuntu 18.04 amd64, rspamd 2.5, but with redis as backend. I get this error about 2 or 3 times per hour:

learn error: Unknown statistics error, found when storing data on backend; classifier: bayes

And this is the result of rspamadm configdump classifier.bayes.backend:

2020-04-10 13:13:38 #0(main) lua_ip_from_string: cannot parse ip:
*** Section classifier.bayes.backend ***
"redis"
*** End of section classifier.bayes.backend ***

Edit: the above result was caused by configuration error which I fixed. Now it's correct:

*** Section classifier.bayes.backend ***
"redis"
*** End of section classifier.bayes.backend ***

But I still get learn error: Unknown statistics error, found when storing data on backend; classifier: bayes.

[norbusan]

Just to chime in, same here, too, Debian buster, rspamd 2.5, redis.

2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_controller_check_password: allow unauthorized connection from a trusted IP 127.0.0.1
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_message_parse: loaded message; id: <[email protected]>; queue-id: <undef>; size: 32282; checksum: <5b9b237748ee9acde82ce4219da439af>
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_mime_part_detect_language: detected part language: en
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_mime_part_detect_language: detected part language: en
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_task_process: learn error: Unknown statistics error, found when storing data on backend; classifier: (null)
2020-04-22 19:30:07 #9595(controller) <e3cc9c>; csession; rspamd_controller_learn_fin_task: cannot learn <[email protected]>: Unknown statistics error, found when storing data on backend; classifier: (null)

[bong0]

I finally resolved the issue in my case, though with the now unsupported version 2.2 on debian jessie (8.11). I want to share it nevertheless, it took the following:

1. I removed /etc/rspamd/local.d/statistic.conf, which previously contained

classifier "bayes" {
    servers = "127.0.0.1:6379";
}

1. I introduced /etc/rspamd/local.d/classifier-bayes.conf with the contents:

servers = "127.0.0.1";
backend = "redis";

1. Restart rspamd.

After uploading ham/spam via rspamc or the webinterface, I started seeing heavy output in redis-cli monitor :)

[R3belWithoutAPause]

This fixed my problem running Rspamd 2.5 on Slackware. Thank you!