VictoriaMetrics requires more file descriptors than our system can support, and crashes

[anjakefala]

Describe the bug
Recently, I have run into problems where VictoriaMetrics has crashed, and done frequent restarts. These frequent restarts happen because it has run out of file descriptors, and hit a limit.

The first time this happened, I increased the ulimit -n to 65535, as suggested. Things were fine, for a while, but now it is crashing again.

I noticed that it has more than 10,000 file descriptors opened shortly before it crashes. Yet, upon startup, I only see "88 parts loaded". Why are there so many file descriptors open? How can I constrain this? it currently refuses to start up.

I am going to increase the number of permitted file descriptors, but I am concerned it is going to hit the new limit, and crash again. What are my options?

Expected behavior
VictoriaMetrics should adapt to the number of available file descriptors it has, instead of crashing, and rebooting repeatedly. There should ideally be a way to configure a file descriptor limit.

Version
victoria-metrics-20200714-141946-tags-v1.38.1-0-gb442a42d8

[hagen1778]

Hi @anjakefala!
A lot of nice explained details about file descriptors usage can be found here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618#issuecomment-663281866
Pls note that TCP connections are also using file descriptors in system and could be exhausted by writers/readers connecting to VM.

[anjakefala]

Thanks @hagen1778! That does answer my question.

I bumped available file descriptors. =)