Stk-code: Performance, Polycount and number of objects

[samuncle]

In order to better optimize the engine and the way I do tracks I wanted to have a better picture of what's optimal in term of scene setup for optimal performances. Since I wanted something that can be reproduce and tested by various people under various conditions I decided to create 4 maps to be used as benchmark testing various scenarios.

I'm well aware those benchmark are heavy and I while it doesn't fully reflect our tracks I wanted to see the engine under extreme circumstance to better plan my next maps and make sure they will be optimized.

The test

The test is designed to have 3 tracks with identical geometry and polycount and testing 3 approach.

• The polycount in all case is 2,242,122 triangles (to my knowledge it's highest track ever loaded in the game)
• The appearance, setup, etc is identical

First scenario

The first scenario is meant to test the number of object. Each patch of grass is an independent object using LoD to try to reduce the amount of polygons displayed.
This scenario emulate how maps are made nowadays with objects that use LoD to try to improve performances.

Second scenario

The second track is meant to test if merging all objects into one big object is better or not regarding performances. This would emulate a lazy approach of just having one big object for the whole track.

Third scenario

The third track is a variation of the second approach where all objects everything is merged into a big object. However then the object is split into 4 x 4 roughly identical chunks each of them which is a separate object.

Fourth scenario

The last scenario is an improved version of the third one which add LoD to chunks to try to mitigate the amount of polygons which are drawn on the screen

Hardware / Software setup

• CPU Intel Core I7-3770 @ 3.40 GHz
• Ram: 16 Gb
• Operating System: Windows 10 64 bits
• GPU: Nvidia GTX 1080
• Vram: 8 Gb
• SuperTuxKart dev version (master) built in release mode
• Windowed Mode
• Resolution: 1920*1080
• Everything set to ultra/high except Texture Compression and glow outline which are disabled

Results

Disclaimer

Let me be clear here, I'm aware this scenario is quite extreme both with high end computer that was used and the massive polycount. The objective is to see how the engine and various CPU/GPU handle such cases.
You are welcome to try your own setup and I would like to see more results. The more results we get the better it will be to know how to improve the performance of newer tracks.

Now let's go with results:

First Scenario

The first maps is with 5000 separate grass patch and LoD

• With LoD enabled (to emulate a real case of LoD use)
  On average I got ~70 FPS. Sometime it went down to 40.
• With LoD disabled (to emulate the conditions of other scenarios where no LoD is present)
  On average I got ~65 FPS.
• With aggressive LoD
  On average I got 90 FPS

With shadows disabled it made almost not difference

Second Scenario

In the second scenario we have one big object with the whole geometry in it.

• We can't have LoD with only one object, so it wasn't used
  On average I got ~ 30 FPS

With shadows disabled I have 100 FPS

Third Scenario

In the third scenario we have the scene split into 16 chunks

• LoD was also disabled in the third scenario
  On average I got ~ 60 FPS

With shadows disabled I have 110 FPS

Fourth Scenario

In the last scenario we add LoD to the 16 chunks
On average I got ~100 FPS which is probably the best case. However the popping was quite noticeable.

With shadows disabled I have 100 FPS

How to test yourself?

I published all 3 maps (it's the WiP category and they are called first, second and third scenario). Just run them and see by yourself.

Conclusions

What I think is interesting is the overhead associated with many small objects. The engine doesn't perform well when it's overwhelmed by a lot of objects.

We can also see something interesting without taking into account shadows it's better to merge all objects into one big model. We have almost no performance hit and as soon as we use split chunks we have virtually no performance drop (and still rendering 2200k poly which is quite impressive when you think about it.

One of my first recommendation would be to have an option to disable shadows for some objects especially stuff like grass or to only enable it for the first shadow cascade.

[auriamg]

Your merged benchmark (4) is > 100 MB in size so we would rather not do that, please. We can probably find other ways to make library objects faster, which would be a better solution. It's also the cleaner way in theory due to instancing. We likely have a CPU bottleneck and should fix that instead of using workarounds that mess up meshes

Adding a way to exclude some objects from shadow cascades would be a good idea too

[hiker]

+1: this kind of track does not belong into assets. Best create a stand-alone test repository for things like that!

+1: without a detailed analysis this test does not really tell us anything: you might restrict yourself from doing things, while there might be a simple reason and fix. Plus as you said, the tests do not reflect tracks, so while it might be interesting if you are developing a game engine, we on the other hand are developing a game, so I would prefer to focus on that. Graphics has already received a lot of development time in the last year or so (Benau and Deve), while game play (and steam port) is basically stagnating (partly because I am not finding enough time for networking).

[samuncle]

Hi

Of course I won't merge objects in blender. It's just a test to see how the engine reacts and what would be the optimal setup.

For the stand-alone repository why not, I just wanted an easy way for people to try and give feedback.

[qwertychouskie]

1st: average ~36-40 FPS
2nd: average ~12 FPS
3rd: average ~14-18 FPS
4th: average ~25 FPS

The first one only showed objects close to the kart, so it looked weird, but it performed the best.
The second and third always had all the objects showing, and thus looked the best.
The fourth one looked better than the first, object appeared in chunks and many more were visible on screen.

Intel HD 4000; Linux; Mesa 17.2.4

Irrlicht Engine version 1.8.0
Linux 4.13.0-31-generic #34-Ubuntu SMP Fri Jan 19 16:34:46 UTC 2018 x86_64
[info   ] IrrDriver: OpenGL version: 4.2
[info   ] IrrDriver: OpenGL vendor: Intel Open Source Technology Center
[info   ] IrrDriver: OpenGL renderer: Mesa DRI Intel(R) Ivybridge Mobile 
[info   ] IrrDriver: OpenGL version string: 4.2 (Core Profile) Mesa 17.2.4
[info   ] GLDriver: ARB Buffer Storage Present
[info   ] GLDriver: ARB Compute Shader Present
[info   ] GLDriver: ARB Arrays of Arrays Present
[info   ] GLDriver: ARB Texture Storage Present
[info   ] GLDriver: ARB Texture View Present
[info   ] GLDriver: ARB Image Load Store Present
[info   ] GLDriver: ARB Shader Atomic Counters Present
[info   ] GLDriver: ARB Shader Storage Buffer Object Present
[info   ] GLDriver: EXT Texture Compression S3TC Present
[info   ] GLDriver: ARB Uniform Buffer Object Present
[info   ] GLDriver: ARB Explicit Attrib Location Present
[info   ] GLDriver: EXT Texture Filter Anisotropic Present
[info   ] GLDriver: Geometry Shaders Present
[info   ] GLDriver: ARB Texture Swizzle Present
[info   ] GLDriver: ARB Pixel Buffer Object Present
[info   ] GLDriver: ARB Sampler Objects Present
[info   ] GLDriver: ARB Vertex Type 2_10_10_10_rev Present
[info   ] GLDriver: ARB Instanced Arrays Present
[info   ] SharedGPUObjects: Hardware Skinning enabled, method: TBO, max bones: 1024

[ghost]

I will leave a brief suggestion here, to create a automatic benchmark track, like Counter-Strike: Source's _VIDEO STRESS TEST_, it lasts like 1 minute and it tests a few things like fire, water, etc, and then gives you the average FPS and an opt-in window asking to send the data to developers which include the results and hardware information.

And put this benchmark inside Configurations so anyone could make quick a test.