SuiteCRM fails to adequately housekeep session files potentially resulting in server failure

Created on 18 Feb 2017 · 24Comments · Source: salesagility/SuiteCRM

There may be implications with Custom Session Directory #877 as that exacerbates the problem, but system concerned is unusable and it is not possible to test this..

Issue

session files are not adequately managed. This can result in huge numbers of session files accumulating on the server e.g:

ls -al session_dir | wc

after 6 months use gave:

483419 4350764 41090483

which is over half a million session files. Attempting to remove them:

rm sess_*

bash: /bin/rm: Argument list too long

as by this point the damage had been done as bash relies on temporary files in order to do this, and as no spare inodes were available, the removal process failed. You can do each one individually.....but you can't keep up with the creation rate!

Expected Behavior

SuiteCRM AND the server it is on should not be brought to a halt! SuiteCRM can actually disable everything on the server.

Actual Behavior

All of the inodes on the mounted drive are used up preventing the writing of any data to the drive, preventing any application that needs to write to that drive from functioning correctly. Signs of this happening are:

pje@archimedes:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 4.0G 8.0K 4.0G 1% /dev
tmpfs 802M 244K 802M 1% /run
/dev/xvda2 99G 37G 57G 39% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 4.0G 0 4.0G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/xvda1 232M 82M 138M 38% /boot
pje@archimedes:~$ echo 1 > test.txt
-bash: test.txt: No space left on device

(i.e. you can't write to a drive that clearly has sufficient space on it)

In order to identify that you are affected by this issue, you can:
df -i
and you can find the offending directories with:
find / -xdev -size +100k -type d

Possible Fix

Add to the cron process that a command to housekeep the session folder on a regular basis. There are probably better ways within the framework to carry out housekeeping on the sessions directory.

I have introduced a cron task to delete all of the session files at 1am as a temporary measure.

Steps to Reproduce

Have a suiteCRM installation with a custom folder for session files.
Wait 6 months for all of the session files to accumulate and use up all of the inodes on the storage device. Information above gives an understanding of the end result and points to the severity of the problem.

Context

Server failure

Probably a medium->high issue as it not only brings down the suiteCRM instance but also the rest of the server.

There is also the failure that has occurred not being immediately obvious due to the logging needing to be improved on the operating system itself:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1665833

Your Environment

SuiteCRM Version used:
Multiple versions of suiteCRM on the system, which started as 7.7.5, and have now all been updated to 7.7.8 and 7.7.9
Browser name and version (e.g. Chrome Version 51.0.2704.63 (64-bit)):
Environment name and version (e.g. MySQL, PHP 7):
mySQL/PHP - not important can't create temporary file: No space left on device when trying to diagnose anything!
Operating System and version (e.g Ubuntu 16.04):
Ubuntu 14.04

Important Fix Proposed Bug

Source

anothermouse

👍1

All 24 comments

Current workaround is to add the following to my crontab file:

SuiteCRM kill off session files to prevent accumulation:

0 3 * * * rm -Rf /var/www/vhosts/crm.mysite.com/private/sess* > /dev/null

but this has to be done before there are too many session files, or it will not work (see above). If you hit this situation, the approach I've used to recover the virtual machine was to delete about 20 files manually, then:

ls sessiondir | grep sess_ > myscript.sh

Edit myscript.sh and add "#!/bin/bash" to the top and then run a substitution to insert "rm " in front of every filename. For vi:

:%s/sess_/rm sess_/g

does the job :)

chmod +x myscript.sh

./myscript.sh

If you have screen on your system, you can run it in that, and go and have a cuppa!

Unfortunately this doesn't quite sort out the problem as:
ls -al sessiondir
root@archimedes2:/var/www/vhosts/demo.crm.unipart.digital# ls -al
total 9000
drwxr-xr-x 6 root root 4096 Jan 17 17:28 .
drwxr-xr-x 27 root root 4096 Feb 19 02:03 ..
drwxr-xr-x 17 www-data www-data 4096 Jan 17 17:30 httpdocs
drwxr-xr-x 2 www-data www-data 4096 Jan 17 17:30 logs
drwxr-xr-x 4 root root 4096 Jan 17 17:16 private
drwxr-xr-x 2 www-data www-data 9195520 Feb 19 02:08 sessions

shows a ridiculous size for the sessions folder! Options:

Rename the folder, create a new one and copy any remaining files across, then delete the messed up folder
fsck apparently works as well:
http://stackoverflow.com/questions/653096/how-to-free-inode-usage

anothermouse on 20 Feb 2017

By default, the system removes session files older than about 30 minutes, on average, every 8 hours, more or less. Clearly this is a bug because the system failed to remove them for 6 months.