Javalin: Create simple JMH performance test suite

[tipsy]

It would be nice if we could test the differences in performance between different Javalin versions. We could create a simple test setup like this: https://github.com/orangy/http-benchmarks

Things to include:

• Simple Hello World
• Responses of different lengths (and compressions), both static and dynamic
• Uploads
• Methods that parse information from requests

Things not to include

• WebSockets
• JSON serialization

[vn7n24fzkq]

Hi, I want to help.
But no idea about how to start.
Create another repo for benchmarks ?

[tipsy]

@vn7n24fzkq, a new repo would be fine for this. I could make one in the javalin organization?

[vn7n24fzkq]

Sure, that sounds great.

[tipsy]

@vn7n24fzkq, I've given you write access to https://github.com/javalin/performance

[tipsy]

I suggest you start by just copying https://github.com/orangy/http-benchmarks, but instead of different frameworks you add different versions of Javalin. This will cover the "Hello World" type test, then we can build on that later.

[vn7n24fzkq]

OK, I will do that.

[vn7n24fzkq]

I am already add benchmarks suite.
And I need some advice on different version of javalin in maven.

[vn7n24fzkq]

I am trying to use maven-shade-plugin as a solution, but not sure it will be good idea.

[tipsy]

Not sure if this is helpful: https://dzone.com/articles/java-classloader-handling

[tipsy]

If you can shade the different versions to contain the version names, that should be okay 🤔

[tipsy]

How is this going @vn7n24fzkq ?

[vn7n24fzkq]

I am sorry that I didn't reply in time due to the busy work in recently.
I have tried classloader and it could be a solution, but this is means I need pre-download different version of javalin in repo.
I am not sure about that pre-download is right way.

[tipsy]

I don't think there's a right or wrong way here. Adding 10+ different versions of the same library to a project is always going to be "wrong", so whatever works is fine. Classloader, shading, multiple modules, it doesn't really matter, as long as you're able to run the different versions independent of one another.

[vn7n24fzkq]

Got it , I will do it as soon as possible.

[vn7n24fzkq]

I am already create a simple benchmark for version 1.0.0 , but no idea about how to create a anonymous class with classloader 🤔

[vn7n24fzkq]

Cuz create a route path need add a anonymous Handler class

[tipsy]

@vn7n24fzkq How is this going?

[vn7n24fzkq]

Sorry I still not found any way for create anonymous class in class loader,maybe the better way is create JMH for each version with same test methods in repo ?

[tipsy]

Sorry @vn7n24fzkq, I must have forgotten to reply. If you make them all different modules, then I guess they can have completely independent dependency trees. Could you try that?

[vn7n24fzkq]

Sure.

[vn7n24fzkq]

@tipsy I am already make JMH can run with different version of Javalin via command

./gradlew -PjavalinVersion=3.0.0 run

Current version contain 1.0.0 and 3.0.0 .
How do you think ?

[tipsy]

Cool, let me try it!

[tipsy]

Nice work, running it for one version works great! Are you able to create comparisons between different versions?

[vn7n24fzkq]

Sure.
But I can't run different version together at same time.
I can write a script like linux sh or window batch to execute all version benchmarks if needed.

[tipsy]

It would be cool if we could define a base version and run comparisons against that, and get report like "3% improvement in requests per second" 🤔

[vn7n24fzkq]

@tipsy
Sorry about the delay, I was add some commit.
Now you can run benchmark with command.

./gradlew benchmark -PjavalinVersion=1.0.0

./gradlew benchmark -PjavalinVersion=3.0.0

After two benchmark, you can compare those results.

./gradlew compare -Pbaseline=1.0.0 -PjavalinVersion=3.0.0

It looks like.

I also update README.md.

[tipsy]

Very cool! I'll try it out this weekend.

[tipsy]

@vn7n24fzkq I managed to run it for 1.0.0 and 3.0.0, but when I try to run it for 3.4.0 and above I get the following error:

> gradlew build

> Task :kaptKotlin
w: [kapt] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: org.openjdk.jmh.generators.BenchmarkProcessor (NON_INCREMENTAL).

> Task :compileKotlin FAILED
e: C:\Users\David\GitHub\performance\src\main\kotlin\javalin\performance\Benchmarks.kt: (30, 13): Unresolved reference: JavalinBenchmark

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileKotlin'.
> Compilation error. See log for more details

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 3s
3 actionable tasks: 3 executed

Do you know why this happens?

[vn7n24fzkq]

@tipsy It's my fault. Because currently the benchmark only be able to run with version 1.0.0, 2.0.0, 3.0.0, 3.1.0, 3.2.0 and 3.3.0.

[vn7n24fzkq]

I will add other versions later.

[tipsy]

Ah, I didn't notice that part! I can also add the rest of the versions :)

[vn7n24fzkq]

Sounds great 😄
I can also do it at tomorrow.

[vn7n24fzkq]

@tipsy Hi, I was added the rest of the versions.

[tipsy]

@vn7n24fzkq Great, thank you!

[tipsy]

I think this can be closed now, thanks a lot for your great work @vn7n24fzkq !