rabbitmqctl hipe_compile stopped working in 3.7.15

[marek-obuchowicz]

We have been using hipe compile command, which worked in previous versions (until rabbitmq:3.7.14-management-alpine). In last release (rabbitmq:3.7.15-management-alpine) hipe compilation fails with:

 > [2/2] RUN rabbitmqctl hipe_compile /rabbitmq_hipe:
#5 1.102 Will pre-compile RabbitMQ server modules with HiPE to /rabbitmq_hipe ...
#5 1.146 Error:
#5 1.146 HiPE compilation is not supported

I'm not sure if this feature was planned to be removed, or is it supposed to work?

[wglambert]

https://github.com/docker-library/rabbitmq/pull/340
https://github.com/docker-library/rabbitmq/issues/336#issuecomment-497871629

[michaelklishin]

This has nothing to do with RabbitMQ version. HiPE is not available in the Erlang build, by design (its future is not at all clear). I'd say relying on HiPE at the moment is not very wise. Hopefully a new LLVM-based JIT will be introduced in Erlang 23 (but we've been waiting for it for years now) to compensate for HiPE falling behind the general VM and compiler development.

[tianon]

Closing given that HiPE support was intentionally removed from this image (and hopefully the additional context for why provided here is helpful). :+1: :heart:

[gerhard]

Even though this is closed now, I would like to know your reasons @marek-obuchowicz for using HiPE. Care to share?

[marek-obuchowicz]

Sure...
We're using rabbitmq in docker for Spryker eCommerce platform. The whole setup is pretty resource-hungry (several web apps, sql, queue, es, redis, jenkins, proxies). We have noticed that this setup occasionally crashes for developers running docker-for-mac/win, where memory/CPU is usually constrained. Some of the jobs were failing to connect to rabbitmq under heavy system load. The team tried several possible improvements and discovered that running rabbitmq with hipe works faster and actually mitigates issues caused by overload. Either it causes less load itself or behaves better on a system with loaded CPU and a little free memory... Since switching to HiPE (which we precompile and distribute in docker image for fast startup) seems like a working solution for the project (at the moment in dev/test stage only, not used yet in production). As this project doesn't rely on newest rabbitmq features, we will pin container version to 3.7.14.

[gerhard]

This is great feedback, thank you.

I am suspecting that your Erlang VM is sharing CPUs with other processes that run in other containers. Can you confirm?

If you do not dedicate CPUs to Erlang, and do not configure schedulers to utilise the dedicated CPUs, the runtime cannot make any of the optimisations that are required for the node to operate efficiently.

The reason why HiPE works better for you is because scheduling of Erlang processes is done differently and the runtime is less sensitive to CPU sharing.

Can you wrap your workload in docker-compose and share it? I'm thinking PerfTest to simulate the number of connections / queues / ingress & egress messages, etc. This docker-compose.yml has all the building blocks that you need to put something together quickly.

If we can prove via metrics that HiPE makes a significant enough difference, and https://github.com/erlang/otp/pull/2262 is merged and released, we might re-consider adding it back into this Docker image.

I would also like to ask for more clarity from the OTP team re HiPE. All release notes & docs read as if HiPE is a partially supported feature, but it's not bad enough to remove it, so 🤷‍♂ . It's difficult to have strong opinions about a half-full / half-empty glass.

[marek-obuchowicz]

Indeed, we have the situation where many processes are sharing CPU's. We have PHP, two JVM-based apps (jenkins, elasticsearch), redis, postgresql database, erlang-based rabbitmq, some nginx processes - all running on (at least) two shared CPU cores on development environments - on Mac/Win it means it runs inside docker VM. This is definitely the case where many processes have to "fight" for CPU resources. Your explanation about CPU scheduling done different with/without HiPE seems to be very logical.

Gerhard, I will be happy to share docker-compose with you: https://gist.github.com/marek-obuchowicz/fe58551e36866be22561644b2a092606

[marek-obuchowicz]

I will try to get benchmarked data for you in our env. Not sure when I will be able to do that, as there is some pressure for other topics for now - but i'll be happy to provide you as much data as possible.