Orientdb: Error "java.net.SocketException: Too many open files" when create class

[thanhphi0401]

OrientDB Version: 3.0.25

Java Version: 8

OS: Ubuntu 18.10

Expected behavior

Created success, not raise any error

Actual behavior

java.net.SocketException: Too many open files

Steps to reproduce

My OrientDB system has over 100 classes with over 50GB data, setup in cluster mode with 6 node.

1. Open console (run ./console) and login to server....
2. Run command "create class ..." -> nothing problem
3. Run command "create property ..." -> nothing problem
4. Run command "create index....." -> nothing problem
   ..........................
5. Run command to create other class : "create class gameapp_platform extends V", it has been error .
   Detail error:
   Error on client connection java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at com.orientechnologies.orient.server.network.OServerNetworkListener.run(OServerNetworkListener.java:188)
   I've tried create again but still has this error. Cluster seem to be failed.
   I've set ulimit open file for linux but seem to be not working.

Please help me.
Thank you

[laa]

Hi, do you have several nodes installed on the same server?

[quanghaisoft]

only one node on one server/IP

[quanghaisoft]

[quanghaisoft]

cluster mode with 6 node on 6 server.but if i drop some class ,the error disappear

[laa]

I see how many connections do you have? Could you send me the output of the server during the start?

[thanhphi0401]

I see how many connections do you have? Could you send me the output of the server during the start?

you can see, about 56 connections now, some time up to 200 connections .

[thanhphi0401]

I see how many connections do you have? Could you send me the output of the server during the start?

after start success, this is status.

[laa]

Send me whole server log just from the begging

[thanhphi0401]

Send me whole server log just from the begging

5000 line logs, can't send all :)
i will send you what main infomation

Do you need whole log file, i will send for you?

[laa]

OK, I have seen what I need, could you send me the current limit of open files inside of OS? Do you use container inside of the cloud?

[thanhphi0401]

OK, I have seen what I need, could you send me the current limit of open files inside of OS? Do you use container inside of the cloud?

No i don't use docker.
This is current limit

[laa]

You see you have size 65K open files but database for som reason detects a bigger amount of open files. What you need is to change open files limit in DB to the 32K open files. Using this system property -Dstorage.openFiles.limit=32000

[thanhphi0401]

it's mean "Dstorage.openFiles.limit" will always set smaller than system limit?

[laa]

yes sure, you need to have room to open more files for "system" needs too.

[thanhphi0401]

yes sure, you need to have room to open more files for "system" needs too.

i want to ask more: Do i need to add " LimitNOFILE="number limit" " to .service file? (because i run orientDB as service on Linux)
Like this

[quanghaisoft]

You see you have size 65K open files but database for som reason detects a bigger amount of open files. What you need is to change open files limit in DB to the 32K open files. Using this system property -Dstorage.openFiles.limit=32000

OK thanks you very much!

[thanhphi0401]

You see you have size 65K open files but database for som reason detects a bigger amount of open files. What you need is to change open files limit in DB to the 32K open files. Using this system property -Dstorage.openFiles.limit=32000

thank you too much for the good solution for us <3