Crystal: About Crystal Benchmarks

[kazzkiq]

It seems TechEmpower benchmarks already have a Crystal implementation running in it, but surprisingly Crystal couldn't finish the tests.

Aditionally, the benchmarks from Kemal (a Crystal framework) all ran extremely bellow average in the reports, which got me surprised as in my machine the req/s was generally way higher than any other competitor (Node.js, PHP, Elixir/Phoenix, etc).

Do you guys have any idea as to why Crystal "breaks" in TechEmpower benchmarks? And why Kemal seems so slow in their benchmark compared to other frameworks running on slower languages?

Could this be just a bad implementation in the benchmark example, or maybe a bug in the language? I can't believe Crystal couldn't even complete the benchmark there, while in my machine it beats the crap out of every other language/framework I've tested.

I think that even due to marketing concerns, perhaps we could try to address and "fix" the TechEmpower code for newcomers.

[refi64]

IIRC one of the reasons was that most of the others had actual multithreading, whereas Crystal is currently only single-threaded and just concurrent (the terminology here is enough to confuse Tesla, I swear...).

Oh, and s/bellow/below.

[drosehn]

Hmm. That page with benchmarks has one entry for "kemal (postgresql)" and for "kemal (redis)", and it's the benchmark for the second entry which did not finish. The first one did quite well. Both list crystal as the language, but I do not do web development, so I don't know what the difference is between those two entries.

[jwoertink]

Node is showing as tons faster, but node is single threaded. I'm calling shenanigans here... How is express blazing faster than Goji?

[refi64]

One of the Kemal PostgreSQL ones got a 0 in... everything. That doesn't seem right...

[RX14]

There's 2 issues here, broken results and slow results. The slow results are likely caused by parallelism. The server the benchmarks are running on has something like 48 cores but each core is a bit shit by itself. Nodejs uses the cluster module to make itself multi threaded, I suggest we write a little wrapper shell script to launch 48 copies of the crystal binary with reuseport enabled.

[RX14]

In fact, I'll look into this more and try to make a pr when I get home.

[mverzilli]

This is probably not the best place to track the problem, but given how much exposure the TechEmpower benchmarks have, we can give them some special treatment :).

[kazzkiq]

In case anyone is interested in the code used in benchmarks, you can find it here.

The Crystal code for both JSON and Plaintext tests can be found here and its extremely simple. Here is the part that seems to do the magic:

when "/json"
    response.status_code = 200
    response.headers["Content-Type"] = "application/json"
    response.print({message: "Hello, World!"}.to_json)
when "/plaintext"
    response.status_code = 200
    response.headers["Content-Type"] = "text/plain"
    response.print "Hello, World!"
else
    response.status_code = 404
end

The above code, which is responsible for both tests, didn't completed the benchmarking (TechEmpower don't say exactly why they failed).

[RX14]

I'm looking into this now.

[akzhan]

Take a look at https://github.com/TechEmpower/FrameworkBenchmarks/pull/2761.

It passes all tests on my vagrant box.

tfb --mode verify --test crystal-raw
tfb --mode verify --test kemal

[RX14]

@akzhan sorry, I should have posted an update here so we didn't duplicate effort.

[akzhan]

@RX14 yes, I found your pull request later (https://github.com/TechEmpower/FrameworkBenchmarks/pull/2758).

For me it was a curious first experience in studying crystal.

[luislavena]

The Crystal code for both JSON and Plaintext tests can be found here and its extremely simple. Here is the part that seems to do the magic:

Not sure why it didn't complete, however this code:

response.print({message: "Hello, World!"}.to_json)

Ends allocating a NamedTuple, then converting into a string, which then is written into the response. I would recommend changing it to:

{message: "Hello, World!"}.to_json(response)

Which will avoid the intermediate string, reducing possible stress to the GC.

[RX14]

@luislavena In the latest round 14 previews crystal (raw and kemal) did complete in all tests, so it does work. It's now simply a performance problem, and that's what my PR addresses.

[kbrock]

thanks @RX14

[asterite]

Should be discussed in Google Groups.