Rabbitmq: Erlang 22 series support

[michaelklishin]

This is not an issue but rather a heads-up and a place to discuss how we want to support Erlang 22.x which is now GA.

This image currently provides a single (latest) version of Erlang. If we want to continue this way, should we wait for a few 22.0.x patch releases? Should we consider supporting two major versions?

@gerhard

[gerhard]

This image currently provides a single (latest) version of Erlang

As it currently stands, Erlang/OTP versions are already pinned to v21 and follow the official Which Erlang - Max supported version. This feels right from all perspectives.

Should we wait for a few 22.0.x patch releases?

Yes. Out of experience, it's always best to wait out at least 2 patches for a new major release before upgrading, especially production.

Should we consider supporting two major versions?

I don't see sufficient benefits in supporting multiple major versions. A single Erlang/OTP version feels adequate.

The plan for Erlang/OTP 22 is to start battle-testing it as early as next week. Most of this will be via rabbitmq-prometheus. The new metrics are insightful in more than one way, I am confident that we have everything we need to make the right decisions. FWIW, the Dockerfile is already based on the official Ubuntu one, it's an easy way of sharing relevant learnings.

Any thoughts @tianon & @yosifkit ?

[tianon]

Sounds reasonable! We'll look forward to the results of your testing. :+1:

[gerhard]

This week was busier than usual, will follow-up next week. 🌆

[gerhard]

The only nit-pick that I've come across in OTP 22 this week is ERL-959. It is possible that this yet-to-be-acknowledged bug is present in prior versions (e.g. 21, 20, etc.), I wouldn't worry too much about it in the context of the Docker image.

Considering that we are now publishing OTP 22 packages for both deb & rpm, as well as officially supporting OTP 22.0.x since 3.7.15, it's safe to update the OTP pinning to 22 in all Docker images 👍

I plan to continue with OTP 22 next week, will update this thread if anything comes up.

Btw, there are a few good comments on RabbitMQ compatibility with Erlang 22.0 GA in this rabbitmq-users mailing list thread by @michaelklishin

[tianon]

Given the notes about HiPE in that thread, should we also remove our HiPE support from the image?

https://github.com/docker-library/rabbitmq/blob/dbbe820acaac916a975ccca45ea1a3662fd4ac59/Dockerfile-ubuntu.template#L119
https://github.com/docker-library/rabbitmq/blob/dbbe820acaac916a975ccca45ea1a3662fd4ac59/Dockerfile-alpine.template#L114
https://github.com/docker-library/rabbitmq/blob/dbbe820acaac916a975ccca45ea1a3662fd4ac59/docker-entrypoint.sh#L58

[gerhard]

Yes, definitely.

This is from OTP 22 release notes:

In OTP 22, HiPE (the native code compiler) is not fully functional. The reasons for this are:

Search for OTP-15596 in the above notes to see the rest.

Also, in OTP HiPE docs:

HiPE and execution of HiPE compiled code only have limited support by the OTP team at Ericsson. The OTP team only does limited maintenance of HiPE and does not actively develop HiPE. HiPE is mainly supported by the HiPE team at Uppsala University.

[gerhard]

Nice!

Btw, ERL-959 is not a bug. It was a matter of learning something new - see that issue for full context.

[avihil1]

Yes, definitely.

This is from OTP 22 release notes:

Does it mean that RabbitMQ will be slower now, without HiPE compiling ? @gerhard

[gerhard]

In the scenarios that HiPE was making RabbitMQ faster, then yes, RabbitMQ will be slower without HiPE.

In real-world scenarios, disks, network contention or lower-frequency CPUs (e.g. AMD Epyc) will have a higher impact on RabbitMQ than HiPE did. Client configuration (e.g. publisher confirms, multi confirm, etc.) will also have a significant impact on RabbitMQ's perceived speed.

It would be interesting to know what impact (if any) disabling HiPE in your production has (everyone's production is different). Care to share the performance that you are expecting out of RabbitMQ @avihil1?

Erlang Jit that is expected to ship in OTP 24 (summer 2021) - https://github.com/erlang/otp/pull/2745 - is the HiPE replacement that I am looking forward to 😉

You may find https://www.rabbitmq.com/blog/2020/06/04/how-to-run-benchmarks/ & https://github.com/rabbitmq/workloads useful. There are some good recent posts on our blog on this topic: https://www.rabbitmq.com/blog/category/performance-2/

[avihil1]

Thanks !

We didn't benchmark our production, we had issues in the past, wanted to make sure that if we upgrade to non-HiPE version, issues won't pop up

[gerhard]

My recommendation would be to not upgrade until you are satisfied with the benchmark results.

Don't benchmark production. Setup an identical environment and simulate the traffic that production is expected to handle. It doesn't get any easier than this: https://github.com/rabbitmq/tgir/blob/S01E07/s01/e07/k8s/rabbitmq/default-metrics.yml. For extra context: https://github.com/rabbitmq/tgir/tree/S01E07/s01/e07

Everything that I mentioned above is from years of experience with under-performing production RabbitMQ deployments. I have more specifics in this RabbitMQ Summit 2018 talk: Real-world RabbitMQ deployments

[avihil1]

Thanks! very helpful