Hhvm: /var/lib/hhvm/sessions has millions of files

[gabu69]

/var/lib/hhvm/sessions is getting millions of files filled in and its not being cleaned, i have other 2 servers with the same config as this server and they have lets say hundreds of files

But in my case last check i had 3,182,200 files in the folder

What could it be?

[creativeprogramming]

Did you tried to setup session.gc_maxlifetime, session.gc-probability, and session.gc-divisor in php.ini?

http://php.net/manual/en/session.configuration.php#ini.session.gc-probability

[pjv]

one solution for ubuntu (could be adapted to other distros) here: http://community.rtcamp.com/t/hhvm-session-files/3639

[fireflydev]

We are having the same problem on HHVM 3.8.0 on Ubuntu 14.04.3 LTS. On one server in particular we accumulate millions of files in /var/lib/hhvm/sessions, we then run out of inodes and the server crashes. Offhand, I note that this particular server/site does use ini_set('session.gc_maxlifetime', 86400); in the code. Perhaps this triggers the condition for HHVM.

[RavanH]

Same issue here on a Debian 8 with 2,985,426 files in /var/lib/hhvm/sessions ... have session.gc_maxlifetime = 1440 in /etc/hhvm/php.ini but that does not seem to have prevented all inodes on my server being filled up !

[ahmetozer]

Ubuntu LTS 14.04
1086656 file created and did not write data to the server disk.
"Session.gc_maxlifetime = 1440" is available in the configuration file, but does not work apparently

[Bogdaan]

ubuntu server: gc_maxlifetime not works, and create milions inodes

[RavanH]

This issue is still labelled "needs more info" ... but what info is needed?

The OP on issue #5034 sais "It seams that the garbage collection is not working." ... Is this related even though "session.save_handler = memcached" there?

[rbongerswux]

Same issue here on a Debian 8 with 2,985,426 files in /var/lib/hhvm/sessions ... have session.gc_maxlifetime = 1440 in /etc/hhvm/php.ini but that does not seem to have prevented all inodes on my server being filled up !

[RalphCorderoy]

Trying to delete 3.5e6 old session inodes from a single directory doesn't go well with some types of filesystem. I had to pace it every few seconds over a week to avoid impacting the machine's behaviour.

[ahmetozer]

@RalphCorderoy I recommend php7

[Firegarden]

Wakey wakey boys and girl,
This issue STILL exists. The painful part of this issue is that it will use up all your disk space. So lets consider this problem 'terminal' right?

[Firegarden]

though "session.save_handler = memcached" there?

No this is not related to the memcached setting - I have the same issue without this setting.

Did anyone actually check if session clean up is implemented in HHVM? Sorry to ask but a previous comment said

I'm not entirely sure what gc() is supposed to do on this class but it looks unimplemented.

hhvm/hphp/runtime/ext/memcache/ext_memcache.php Lines 755 to 757 in 8911447 public function gc($maxLifetime) { return true; }

[Rom4eg]

Simple workaround

1. create file /var/lib/hhvm/maxlifetime with content

#!/bin/sh -e

min=1440
for ini in /etc/hhvm/php.ini ; do
    cur=$(sed -n -e 's/^[[:space:]]*session.gc_maxlifetime[[:space:]]*=[[:space:]]*\([0-9]\+\).*$/\1/p' $ini 2>/dev/null || true);
    [ -z "$cur" ] && cur=0
    [ "$cur" -gt "$min" ] && min=$cur
done

echo $(($min/60))

exit 0

this will find and print session.gc_maxlifetime variable

1. create file /var/lib/hhvm/sessionclean with content

#!/bin/sh

[ -x /usr/bin/lsof ] && /usr/bin/lsof -w -l +d "${1}" | awk -- '{ if (NR > 1) { print $9; } }' | xargs -i touch -c {}
find "${1}" -depth -mindepth 1 -maxdepth 1 -ignore_readdir_race -type f -cmin "+${2}" -delete

this will delete all files that was changed older than session.gc_maxlifetime seconds ago

1. make them executable

chmod 755 /var/lib/hhvm/maxlifetime
chmod 755 /var/lib/hhvm/sessionclean

1. create cronjob for root user. I'm just create new file /etc/cron.d/hhvm with content

30 0 * * * root [ -x /var/lib/hhvm/maxlifetime ] && [ -x /var/lib/hhvm/sessionclean ] && /var/lib/hhvm/sessionclean /var/lib/hhvm/sessions $(/var/lib/hhvm/maxlifetime)

start cleaning at 0:30

Tested in Ubuntu 16.04.2 LTS

P.S. Correct me if i'm wrong.

[RalphCorderoy]

Just been asked to fix a server that wasn't functioning and it was again due to inode exhaustion due to 4,000,000 session files in the one HHVM session directory. Seems irresponsible to ship software that creates this ticking failure mode on every installation.

[yohgaki]

I'm just curious. I don't use HHVM daily basis, but I suppose session works just like PHP.

Any of you use path spread option? i.e. Something like

session.save_path = "4;/var/lib/hhvm/sessions"

When path spreading option is used, session module will not remove old sessions as documented.
http://php.net/manual/en/session.configuration.php#ini.session.save-path
Users are supposed to remove old session data files by themselves via cron/etc.

If you are removing session data files correctly, you probably have too long session lifetime. It is easy for crackers to make too many session files. e.g. "ab -c 1 -n 10000000 http://yoursite.example.com/" would create 10000000 session data files.

Session management best practices:

1. Keep session lifetime as short as possible.
2. Implement auto login feature by yourself if automatic login is required. Never use longer session lifetime.
3. Cleanup obsolete session data by yourself if it is needed. Files save handler is one of them.
4. Manage session timestamp by yourself. i.e. You should manage session timestamp by yourself. This is described in this RFC. NOTE: This feature is not implemented in session module since the RFC is declined.

In short, I suppose this is not a HHVM bug.

[RalphCorderoy]

This is a HHVM bug. We don't use path spreading. Sessions only work 'just like PHP' if their parts have been implemented.

[lexidor]

I am going over old issues on this repository, to see which ones apply to the current versions of hhvm.

This issue relates to $_SESSION, which has been removed from hhvm in version 4.4.0.