Rabbitmq: 3.7-alpine-management, is it really stateless and immutable?

[MangoCats]

I've had a problem with my 3.7-management-alpine container which manifested as clients not being able to connect to the Rabbit server (something about an error 541 I think... I wish I had paid more attention). I'm pretty sure it came about as a result of the host system's disk getting full, but what bothered me was that the problem persisted, long after plenty of disk space was freed up, and more worryingly: after the system had been shut down and restarted.

When I killed the rabbit's docker process and removed the container, then re-ran it from the image, it was fine again. I just expected it to "heal itself" and restart from the image when the system was restarted... Is there some option I'm setting wrong to cause this "memory" behavior in the container?

I'm hosting the container on an Ubuntu 18.04 system with docker setup as follows:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y ca-certificates
sudo apt-get install -y curl
sudo apt-get install -y software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

sudo apt-get update
sudo apt-get install -y docker-ce
sudo mkdir -p /etc/docker
sudo cp daemon.json /etc/docker/daemon.json

And, we've added MQTT to the image with this Dockerfile:

FROM rabbitmq:3.7.8-management-alpine
RUN rabbitmq-plugins enable --offline rabbitmq_mqtt
EXPOSE 5672/tcp
EXPOSE 15672/tcp
EXPOSE 1883/tcp
ENV RABBITMQ_DEFAULT_USER OurUser
ENV RABBITMQ_DEFAULT_PASS OurPass
COPY rabbitmq.conf /etc/rabbitmq/

and this rabbitmq.conf:

listeners.tcp.default = 5672
mqtt.default_user = OurUser
mqtt.default_pass = OurPass
mqtt.allow_anonymous = true
mqtt.vhost = /
mqtt.exchange = Our_Exchange
mqtt.subscription_ttl = 86400000
mqtt.prefetch = 10
mqtt.listeners.ssl = none
mqtt.listeners.tcp.default = 1883
mqtt.tcp_listen_options.backlog = 128
mqtt.tcp_listen_options.nodelay = true

The image is built and container run with these commands, once, then it automatically restarts when the system restarts:

sudo docker build -t rabbit37mqtt .
sudo docker run -d --restart always --hostname our-rabbit --name rabbitmqps -e RABBITMQ_DEFAULT_USER=OurUser -e
RABBITMQ_DEFAULT_PASS=OurPass --network=host rabbit37mqtt

I'm hoping for a "stateless and immutable" RabbitMQ server that restores itself to the exact same configuration every time the system restarts... I can't see why that's not happening.

[wglambert]

You haven't declared a volume so the storage for that container is ephemeral
https://docs.docker.com/storage/volumes/

You can either make a Docker-managed volume with docker volume create rabbit-data and then include docker run -v rabbit-data:/var/lib/rabbitmq, or use a directory on your host to mount instead.

Creating the container and adding an exchange:

$ docker run --rm -dit -v rabbit-data:/var/lib/rabbitmq --name rabbitmq --hostname=rabid -p 15672:15672 -e RABBITMQ_DEFAULT_USER=rabid -e RABBITMQ_DEFAULT_PASS=pass -e RABBITMQ_DEFAULT_VHOST=rabidemqueue rabbitmq:3.7.8-management-alpine
6d066daab0844f81f0915626ca1d7d77e12881435cfa5eb75bb773733579dd11

$ curl -u rabid:pass http://localhost:15672/api/vhosts
[{"cluster_state":{"rabbit@rabid":"running"},"name":"rabidemqueue","tracing":false}]

$ docker exec rabbitmq rabbitmqadmin --vhost=rabidemqueue -u rabid -p pass declare exchange name=rabid type=fanout
exchange declared

$ docker exec rabbitmq rabbitmqctl list_exchanges -p rabidemqueue
Listing exchanges for vhost rabidemqueue ...
amq.match       headers
amq.headers     headers
amq.topic       topic
rabid   fanout
amq.direct      direct
amq.rabbitmq.trace      topic
        direct
amq.fanout      fanout

Removing the container and start a new one with the same volume

$ docker rm -f rabbitmq 
rabbitmq

$ docker run --rm -dit -v rabbit-data:/var/lib/rabbitmq --name rabbitmq --hostname=rabid -p 15672:15672 -e RABBITMQ_DEFAULT_USER=rabid -e RABBITMQ_DEFAULT_PASS=pass -e RABBITMQ_DEFAULT_VHOST=rabidemqueue rabbitmq:3.7.8-management-alpine
688d0b4e4fb49d0b261ca438388494d9d07d11a53a990f34a25badd0bffed66e

$ docker exec rabbitmq rabbitmqctl list_exchanges -p rabidemqueue
Listing exchanges for vhost rabidemqueue ...
amq.match       headers
amq.headers     headers
amq.topic       topic
rabid   fanout
amq.direct      direct
amq.rabbitmq.trace      topic
        direct
amq.fanout      fanout

$ curl -u rabid:pass http://localhost:15672/api/vhosts[{"cluster_state":{"rabbit@rabid":"running"},"name":"rabidemqueue","tracing":false}]

[michaelklishin]

RabbitMQ is a data service, those are not "stateless" by definition. Currently we highly recommend decommissioning nodes that have run out of disk space. They are not guaranteed to be able to proceed. We hope that it would be much more predictable in 4.0 or in 4.x.