Rspamd: High memory usage since upgrade to 1.7.5

[ashceryth]

Classification (Please choose one option):

• [ ] Crash/Hang/Data loss
• [ ] WebUI/Usability
• [x] Serious bug
• [ ] Other bug
• [ ] Feature
• [ ] Enhancement

Reproducibility (Please choose one option):

• [X] Always
• [ ] Sometimes
• [ ] Rarely
• [ ] Unable
• [ ] I didn’t try
• [ ] Not applicable

Rspamd version:

1.7.5

Operation system, CPU, memory and environment:

Debian Stretch, 2GB RAM

Description (Please provide a descriptive summary of the issue):

Since upgrading from 1.7.4 to 1.7.5 there is a significantly higher memory usage on my server. The Rspamd processes are using up almost all available memory:

# ps aux --sort -rss
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
_rspamd    772  0.4 28.8 833004 589332 ?       SL   10:24   0:07 rspamd: rspamd_proxy process (localhost:11332)
_rspamd    774  0.3 28.2 832868 576432 ?       S    10:24   0:07 rspamd: normal process (localhost:11333)
_rspamd    773  0.7 18.1 836124 371240 ?       SL   10:24   0:13 rspamd: controller process (localhost:11334)
...

The graph shows that the memory usage nearly doubled after the update:

Please tell me if you need further information!

Compile errors (if any):

Steps to reproduce:

Expected results:

Actual results:

Debugging information (see details here):

Configuration:

Additional information:

[vstakhov]

Disable phishtank.

[ashceryth]

Thanks a lot, that solved the issue:

# ps aux --sort -rss
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
_rspamd   1022  0.3  2.9 266776 60252 ?        SL   11:20   0:00 rspamd: rspamd_proxy process (localhost:11332)
_rspamd   1023  0.6  2.7 267428 56780 ?        SL   11:20   0:00 rspamd: controller process (localhost:11334)
_rspamd   1024  0.3  2.7 266640 56316 ?        S    11:20   0:00 rspamd: normal process (localhost:11333)
_rspamd   1016  0.6  2.5 260784 53048 ?        Ss   11:20   0:00 rspamd: main process
_rspamd   1025  0.0  2.0 260784 41236 ?        S    11:20   0:00 rspamd: hs_helper process
...

[pkalemba]

@vstakhov can you describe why phishtank module is using soo much memory? in my case ~20 GB per node

[vstakhov]

It is a very large hash table that lives in Lua. Unfortunately, that's how
they organise that data. In theory, I could reduce memory usage
significantly but that's definitely not among my high priority tasks. So
I'd recommend not to use it.

On 22 May 2018 10:51:41 Paweł Kalemba [e-mail] wrote:

@vstakhov can you describe why phishtank module is using soo much memory?
in my case ~20 GB per node

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/vstakhov/rspamd/issues/2254#issuecomment-390931973

[pkalemba]

@vstakhov thank you , @ashceryth i think you can close issue yes?

[ashceryth]

@vstakhov Thank you for the elaboration.

[ralphM78]

@vstakhov I have a similar problem with 1.7.6-3~xenial. If the server boots up, everything works fine for days/weeks until "systemctl restart rspamd". After restarting rspamd it takes about 2 minutes, then the server starts swapping. RAM usages increases very fast until the system isn't usable anymore. Disabling phishtank solves the problem, but why this happens only during "systemctl restart rspamd" ? Strange.