Beanstalkd: running out of open files on server causes client connections to hang

[ramSeraph]

Ideally the client connections should be rejected when the server knows it is about to hit the open file descriptors limit.

The limits can be queried by the server using the rlimit apis. or maybe the error code ENOFILE can help decide how to handle it..

Sorry for the lack of details.. will add more when I find time.

[ysmolsky]

This could be solved to make program more robust. What happens if server is about to run out of FDs when it is going to write into binlog? Server should reject connections when there are less than 2 free FDs left, ideally. Not sure if it's possible without sacrificing some performance though.

[silviucpp]

I think an easy workaround is to set the number of file descriptors the app can use to a higher limit when the app starts. This can be done into the code using the rlimit api or inside the systemd scripts using LimitNOFILE=X

If you set a value of 100000 most probably nobody ever will hit this limit with beanstalkd.

[kr]

In the early days of beanstalkd, I did put some effort into making it work properly when it runs out of open fds, but it was tricky and I was never super confident that would be reliable.

Plus, there’s no real solution I’m aware of when it needs to open a new binlog file.

[ysmolsky]

Setting this via rlimit seems like a sane solution.