Orientdb: Recommended CPU and RAM for ~1000 requests/s

[janjilek]

Hello,
I will need to handle 500+ concurrent users per second => ~1000request/s. Right now I have 2 cpu and 15 GB of ram (aws r3.large instance) and I am not able to handle more than 30 request/s (via blazemeter.com or jMeter locally) with simple queries like following: (product contains 7 string properties)

SELECT expand(OUT('HasProduct')[0-99]) FROM Category WHERE id = 'recommended' fetchplan delivery:1

This query is fast, when it's executed only once, but on concurrent connections it's really slow. Doesn't matter if I use differente query (select, insert), I am not able to handle more than 30-50 request/s.

I am using default settings, OrientDB 2.1.16 and OrientJS driver.

Thank you

[wolf4ood]

hi @janjilek

How much does this query takes?

SELECT expand(OUT('HasProduct')[0-99]) FROM Category WHERE id = 'recommended' fetchplan delivery:1

[janjilek]

Elapsed in explain: 26.697037
Explain execution in the web console: 0.072 sec
Query executed in the web console: 0.282 sec

[wolf4ood]

Can you post code you are using for the connection?

[janjilek]

It's strange, when I try it on really slow machine (1 virtual cpu, 512MB ram) i have following:
Elapsed in explain: 25.986715
Explain execution in the web console: 0.07 sec
Query executed in the web console: 0.233 sec

So it looks like some default configuration issue.

[janjilek]

Actually I tried a lot of different approaches:
1) 1 db instance
var server = OrientDB({});
export var db = server.use({});
2) Use of ODatabse
var db = new ODatabase({});
3) Use of Connection pool
var pool = new Pool({
min: 1,
max: 100
});
pool.acquire().then((db) => {});
4) Create N (>=10) connections polls (max = 50) and round robin between them

The option number 4 is the fastest one

[janjilek]

When I execute it on my MacbookPro 13 2015 (i5 2.7, 16GB ram) i have same results.

[laa]

@maggiolo00 probably we should discuss that with @tglman ?

[wolf4ood]

Can you monitor the Cpu of OrientDB while running this test?

[wolf4ood]

how many request with number 4?

[janjilek]

I run jMeter with this setup:
Number of threads (users): 100
Ramp-up period: 1s
Loop count: 5

Results for approach 1:
Samples: 500
Throughput: 7.7requests/s
Min: 235ms
Max: 13314ms
Average: 11579ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3736m 936m 16m S 16.0 6.1 2:40.68 java

Results for approach 3:
Samples: 500
Throughput: 23.6requests/s
Min: 206ms
Max: 4616ms
Average: 3735ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3528m 856m 16m S 66.9 5.6 4:31.35 java

Results for approach 4:
Samples: 500
Throughput: 35.6requests/s
Min: 210ms
Max: 4577ms
Average: 2098ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3670m 805m 16m S 199.5 5.3 3:00.82 java

[janjilek]

I updated results with different approaches. The approach 1 is normally not so bad, because I have more NodeJS instances. This results are only from 1 backend instance.

[StarpTech]

@janjilek Im very interesting in your approach I also have the same configuration but without high traffic. Good luck. I will try to help you when I find something.

[wolf4ood]

@janjilek

with approach 4 is more than 15%.

This is because with approach 1 yes you will have only one connection opened and all the request are serialized.

[janjilek]

Yes, sure, I had error in my code, because I tested it with more backend instances before, so I had 5+ connections to the DB.

[janjilek]

Is 35.6 requests/s expected performance for 2 CPUs and 12 GB of RAM? Is more important to have more CPU cores? Or how many CPUs and GB of memory would you suggest for that kind of requests/s?
Thank you

[wolf4ood]

@janjilek

do you have numbers with 2 instances?

[StarpTech]

perhaps related cause db switch overhead
https://github.com/orientechnologies/orientjs/issues/169

[janjilek]

I will setup 2 instances and i will write the results here.

[janjilek]

Results for approach 1 with 2 backend instances
Samples: 500
Throughput: 13.9 requests/s
Min: 254 ms
Max: 7471 ms
Average: 6072 ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3898m 984m 16m S 32.3 6.4 6:10.84 java

Results for approach 1 with 4 backend instances
Samples: 500
Throughput: 25.6 requests/s
Min: 222 ms
Max: 4317 ms
Average: 3197 ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4092m 1.0g 16m S 66.2 7.0 6:35.14 java

Results for approach 4:
Basically the same like with 1 instance

[wolf4ood]

@janjilek

strange with 4 instances and approach 4 you should go better.

Which is the size of your custom pool?

[janjilek]

I tried 50 pools with max = 5 in 4 instances => 1000 connections. Now i will try 5 pools with max = 50 and maybe 25 pools with max = 25

[janjilek]

So with 2 instances and 5 pools with max = 50 performance drops to 15 requests/s (don't understend why) and CPU usage is only on 35%. With 50 pools with max=5 the performance is +- the same like before.

With 4 instances and 5 pools with max = 50 performance drops to 30 requests/s and CPU usage is on 50%. With 50 pools with max=5 the performance is +- the same like before.

[janjilek]

I will try to add some CPUs and will write you the result with approach 4.

[janjilek]

Results for approach 4 and db server with 4 CPUs:
1) backend server with 4 instances
Samples: 500
Throughput: 73.1 requests/s
Min: 208 ms
Max: 2350 ms
Average: 1024 ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4938m 872m 16m S 297.0 2.8 2:09.32 java

1) backend server with 1 instance
Samples: 500
Throughput: 56.5 requests/s
Min: 262 ms
Max: 3109 ms
Average: 1428 ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4667m 929m 16m S 256.0 3.0 3:27.61 java

[janjilek]

I have tried now again (5 times) with 4 instances and the average result is about 62 requests/s. So not so big difference.

[janjilek]

With 8 CPU on DB machine:
Samples: 500
Throughput: 75.8 requests/s
Min: 202 ms
Max: 2278 ms
Average: 912 ms

Top commnad on server instance
VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6862m 885m 16m S 217.7 5.9 2:05.43 java

[janjilek]

I will try some more stuff tomorrow. It's not using all the db cores, so maybe it's something wrong with load balancing.

[janjilek]

So, I tried today a lot of instance types for backend and database and best these are the best results.

I was able to handle 181 requests/s with c4.2xlarge (8 CPUs, cpu usage 560%) instance on database and 2x m4.xlarge instances (8 CPUs in totall). I tried also more m4.xlarge instance but the result was same like with 2.

When I descreased number of returned products from 100 to 50, I was able to handle 360 request/s but the cpu usage was again about 560%. So there is probably some aws throughput limit and I have to intestigate it more.

The last try was with 10Gbps instances. 1 c4.8xlarge (36 CPUs, 132 GB RAM) for database and 2 m4.10xlarge (40 CPUs, 124.5 GB RAM). The result was 120 request/s (with 1200% of cpu usage), so there is probably some db configuration error.

Do you have any suggestion (configuration), how to test maximum concurrent performance for different queries? Is normal, that OrientJS driver consumes fully 8 CPUs for 200 requests/s?

PS: t2 instances were bottleneck for yesterday testing.

[wolf4ood]

hi @janjilek

about orientjs a lot of cpu time is consumed on deserializing the results set. In 2.2 we implemented the native serializer/deserializer to gain more speed.

[fabriziofortino]

@janjilek

Not sure if this could be relevant but you might lower latency and jitter on ec2 instances with the right drivers, enabling enhanced network capabilities (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html).

[janjilek]

Thank you, I will try new version of driver next week and enhanced network capabilities sounds also good. Will write you back results next week.

[smolinari]

Interesting @fabriziofortino . You would think the best networking would be standard and not an option.

@janjilek - I also know in the past OrientJS had issues with, I believe it was, marshaling and there was discussion on improving the OrientJS driver. It seems it is why ODB did so badly compared against Arango. That was some time ago too. I didn't follow if there were improvements made to the driver or not. If there wasn't, the performance issues are most likely in the OrientJS driver.

Scott

[janjilek]

@smolinari I can try to implement test case also with Java driver and compare results :-)