Eventstore: High CPU load when using Docker container

[puco]

When running the EventStore in docker as part of a local environment I experience a very high load. This is happening from the very beginning when starting with a clean container. When using Windows version the load is the expected 0.0x%.

Details of environment:

OS: Windows 1803, fully patched
Docker:

# docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:05:28 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:13:46 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Eventstore configuration (running with docker-compose):

version: '3'
services:
  eventstore:
    image: eventstore/eventstore:release-4.1.0
    restart: always
    ports:
    - "32113:32113"
    - "31113:31113"
    environment:
      EVENTSTORE_EXT_HTTP_PORT: 32113
      EVENTSTORE_EXT_TCP_PORT: 31113
      EVENTSTORE_EXT_HTTP_PREFIXES: "http://*:32113/"
      EVENTSTORE_START_STANDARD_PROJECTIONS: "true"
      EVENTSTORE_RUN_PROJECTIONS: "System"
      EVENTSTORE_STATS_PERIOD_SEC: 36000

Performance / docker stats:

# docker stats
CONTAINER ID        NAME                    CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
d2473bf6e8a9        projects_eventstore_1   18.55%              57.59MiB / 1.934GiB   2.91%               718B / 0B           1.55MB / 627kB      19

Log file: https://gist.github.com/puco/fdb9f295c525220de0d74f9a4a5996b0

Windows configuration (config.yaml)

ExtHttpPort: 32113
ExtTcpPort: 31113
ExtHttpPrefixes: "http://*:32113/"
StartStandardProjections: "true"
RunProjections: "All"
StatsPeriodSec: 36000

[mjaric]

Here is interesting founding. Looks like mono or code blocks that are compiled when build is run for linux platform, such build is 10 times underperforming pure .NET Framework build.

Version: Windows Build 4.1.1-hotfix1 vs MacOS build with same code base as used for mentioned version

Hardware:
MacBook Pro 15" late 2017 ... ram is irrelevant since ES is using only top 3GB I saw during write banchmark. There is not cluster so it is single node setup.

Same app is used each time compiled in mono 5.1 and same build is run in windows and in Windows. 10 threads concurrently uses same connection to write 100 events in single transaction. Then we started such app 8 times in total on arbitrary computers around in network.

1. Windows 10, SSD Samsung 840 evo... that can do 540Mbps write speed.

Results:
Each second, EventStore managed to persist 120,000 events on disk. Disk was 100% busy. Don't forget it is samsung 840 evo disk, the slower one. CPU (4 cores/8 threads) was at 40% and eventstore almost constantly used 2.5 GB of ram. Average EventStore response was about 70miliseconds, MAX about 100 and min 13 miliseconds. I bet it could do more with better storage :)

1. MacOS High Sierra, SSD APPLE SSD SM0512L that should do 2TBps (havent checked if this is realy possible, but hey, it can for sure do more than 540MBps, right?)

Results:
with same amount of client writers, it didn't manage to accomplish more than 1000 writes per second but with response latency 2.3 seconds. Disk was like at 5% utilization but what is interesting RAM is growing constantly, CPU was 100%.

BTW. Writing clients were akka.net Actors using shared ES connection directly. 10 actors per single connection.

[mjaric]

Just to add one note here that I will try to prove with another benchmark. Wille ago I was trying different ways of using proto messages while I was bulding elixir client for eventstore. What I found is that it is better not using append to stream, instead one has better performance if he explicitly opens transaction and then send one by one eventdata message instead of bulk. My guess here is that bottleneck for write latency is the fact that batch takes more time to decode on either end since there is on both sides limited pool of protobuf message processors. This also can be proven if this pool of message processors and client pool use append only method with implicit transaction but rather they exchange less messages in single batch. The troughput will vary case by case but obvious facts is that Less bytes you send it will be quicker

[gregoryyoung]

So testing performance on mac I can assure you will give slower results. This is largely due to time not being invested into making OSX highly performant. Let me explain a bit why the investment was not made...

All of our production environments that I know of are either running in linux/windows, we have spent a lot of time on both working on performance. We in fact don't recommend running OSX for a production environment. OSX is only really used on dev machines and as such has not had the time spent on performance analysis as the others (it has always been "does it work in the same way" as the most important metric).

This might be worth creating a ticket for, though I am guessing the work done will be to make it 2-3 times faster not the same level of work that has been done in windows/linux in terms of performance (I am sure you can understand why). If you look closely in source you will also as example find quite a few linux/windows specific paths for performance purposes.

Does this make sense?

If want to look into it a bit let me know, I am happy to remote pair etc with you to see if we can find some easy bottlenecks to work around quickly.

[mjaric]

Thanks Greg,
There is one thing tho that doesn’t fall in case you described, docker evenstore container is run still in virtualization, like in windows, where vm Is linux. I haven’t conducted any banchmarks using docker bit if you are curios its few shell strokes away, so let me know if you are interested.

I don’t see any reson to spend any significant time makin osx case in code since I doubt that anyone is using osx in production servers. But I would like to see osx builds downloadable from eventstore.org website :)

[jen20]

We have shipped OS X builds in the past, but stopped when we stopped being able to get them statically linked. We intend to ship them again as part of #1716.

[mjaric]

I fixed build scripts for OS X here #1723. So far I did not see any unexplained CPU spikes. I hope this helps you to bring back official OSS builds on eventstore.org :)

[shaan1337]

@puco @mjaric
As from version 5.0.0:

1. High cpu usage issues have been fixed in linux builds.
2. we now have macOS downloads available on our downloads page.

I'm closing the issue for now, but please feel free to reopen if you're still facing the same problems.