Charts: Rabbitmq auth admin credentials does not work.

[hansehe]

Which chart:
Rabbitmq helm chart version 7.6.6.

Describe the bug
Setting auth.username and auth.password results in 'login failed' when trying to login to the admin dashboard.
The auto generated credentials works the first time it is deployed. However, you are required to set the same auth credentials to update the helm chart, which leads to 'login failed' again.

In summary, setting auth.username and auth.password does not work.

To Reproduce
Steps to reproduce the behavior:

helm install rabbitmq rabbitmq/rabbitmq -f values.yaml

values.yaml:

persistence:
  enabled: false
  size: 8Gi
replicaCount: 1
auth:
  username: user
  password: <password>
  erlangCookie: <cookie>
extraConfiguration: |-
    mqtt.allow_anonymous  = false
    # 24 hours by default
    mqtt.subscription_ttl = 86400000
    mqtt.prefetch         = 10
metrics:
  enabled: true
ingress:
  enabled: true
  hostname: rabbitmq.localhost
  tls: true
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
extraSecrets:
  load-definition:
    load_definition.json: |
      {
          "users": [
              {
                  "name": "amqp",
                  "password": "<password>",
                  "tags": "amqp-client"
              },
              {
                  "name": "mqtt",
                  "password": "<password>",
                  "tags": "mqtt-client"
              }
          ],
          "vhosts": [
              {
                  "name": "amqp"
              },
              {
                  "name": "mqtt"
              }
          ],
          "permissions": [
              {
                  "user": "amqp",
                  "vhost": "amqp",
                  "configure": ".*",
                  "write": ".*",
                  "read": ".*"
              },
              {
                  "user": "mqtt",
                  "vhost": "mqtt",
                  "configure": ".*",
                  "write": ".*",
                  "read": ".*"
              }
          ]
      }
loadDefinition:
  enabled: true
  existingSecret: load-definition
extraConfiguration: |
  load_definitions = /app/load_definition.json
extraPlugins: "rabbitmq_auth_backend_ldap,rabbitmq_mqtt,rabbitmq_web_mqtt"
service:
  extraPorts: 
    - name: mqtt-tcp
      port: 1883
      targetPort: 1883
    - name: mqtt-web
      port: 15675
      targetPort: 15675

Expected behavior
Successfull login with the given credentials

Version of Helm and Kubernetes:
Helm3 and kubernetes v1.16.5

• Output of helm version:

version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Additional context

[marcosbc]

Hi @hansehe, it seems that your issue is being masked by a failed first-time initialization. Despite disabling persistence, our chart enables emptyDir for the data directory volume (i.e. so it can be accessed by init containers), and when restarted the volume does not get cleaned so the initialization logic thinks RabbitMQ has already been initialized.

Hence, the real error can be found during the first initialization. Enabling image.debug=true you can see the failure when executing rabbitmqctl change_password USER PASSWORD:

 13:58:04.37 DEBUG ==> Changing password for user 'user'...
Changing password for user "user" ...
2020-09-15 13:58:05.200 [warning] <0.1302.0> Failed to change password for user 'user': the user does not exist
Error:
User "user" does not exist
 13:58:05.22 ERROR ==> Couldn't change password for user 'user'.

This seems to be caused of the configuration that is being mounted from your values.yaml file, so there is likely an issue there.

[hansehe]

I experienced the same issue when turning on persistence.
But as a temporary solution i managed to create an administrator with the definitions file:

"users": [
                {
                    "name": "admin",
                    "password": "password",
                    "tags": "administrator"
                },
....

[marcosbc]

That error should also be shown when persistence is enabled like you mention. It looks related to the hardcoded user list in your definition, causing the rabbitmqctl change_password command to fail.

Right now I cannot think of a good way to solve this, as the user list seems hardcoded in the definition and cannot be changed. Maybe allowing to disable the user renaming (meaning the chart's auth credentials wouldn't make sense anymore)? The related logic can be found here.

[hansehe]

I actually think that would be a good idea to allow for disabling default auth credentials, and instead rely on the administrator to define admin users with the definition file.

[marcosbc]

Thanks for the feedback. I agree it could make sense as long as it is something that can be changed via an environment variable (the default behavior should be the same as what we currently have though).

Unfortunately we have a lot on our plate lately, so we have a very limited capacity for such changes. However, we encourage users to submit PRs so these changes can be merged earlier. Would you be up for sending a PR making the needed changes in the bitnami-docker-rabbitmq repository?

[hansehe]

I'll try to have a look at it at least 👍

[unitto1]

I have the same issue with this values:
```rabbitmq:
auth:
username: root
password:
erlangCookie:

memoryHighWatermark:
enabled: true

extraPlugins: ""

loadDefinition:
enabled: true
existingSecret: load-definition

extraConfiguration: |-
log.default.level = warning
log.console.level = warning
disk_free_limit.absolute = 50MB
load_definitions = /app/load_definition.json

extraSecrets:
load-definition:
load_definition.json: |
{
"name": "HA",
"pattern": ".*",
"vhost": "/",
"definition": {
"ha-mode": "all",
"ha-sync-mode": "automatic",
"ha-sync-batch-size": 1
}
}

replicaCount: 3

nodeSelector:
"cloud.google.com/gke-preemptible": "true"

affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: "topology.gke.io/zone"
labelSelector:
matchLabels:
"app.kubernetes.io/name": rabbitmq

resources:
requests:
cpu: 300m
memory: 2Gi
limits:
cpu: 300m
memory: 2Gi

requests:

cpu: 1

memory: 4Gi

limits:

cpu: 1

memory: 4Gi

persistence:
storageClass: ssd

pdb:
create: true
minAvailable: 0
maxUnavailable: 1
```
how can we get the generated password?

rabbitmq secret contains wrong credatials in any case, even when no password is filled in values.

[marcosbc]

@unitto1 The way to get RabbitMQ credentials is shown during the helm install command:

```* Please be patient while the chart is being deployed *

Credentials:

echo "Username      : user"
echo "Password      : $(kubectl get secret --namespace default myrabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"
echo "ErLang Cookie : $(kubectl get secret --namespace default myrabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)"

```

It is recommended to obtain them before re-deploying, or secrets would be re-created (and therefore lost since persisted data uses old credentials).

Also, notice that as mentioned in this issue, when mounting definitions the default credentials are not modifiable via rabbitmqctl change_password, causing the initialization to fail. This is a known issue like reported here, feel free to contribute if you would like to.

[unitto1]

@unitto1 The way to get RabbitMQ credentials is shown during the helm install command:

Credentials:

    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace default myrabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"
    echo "ErLang Cookie : $(kubectl get secret --namespace default myrabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)"

It is recommended to obtain them before re-deploying, or secrets would be re-created (and therefore lost since persisted data uses old credentials).

Also, notice that as mentioned in this issue, when mounting definitions the default credentials are not modifiable via rabbitmqctl change_password, causing the initialization to fail. This is a known issue like reported here, feel free to contribute if you would like to.

as I mentioned before in my case secret contains nonvalid credentials.

[pavars]

Issue is still present in helm chart 7.6.7, doesn't matter how it's installed (as a dependency or standalone)
helm install rp-rabbitmq --set auth.username=rabbitmq,auth.password=rabbitmq,rabbitmq.rabbitmq.username=rabbitmq,rabbitmq.rabbitmq.password=rabbitmq,replicaCount=1 bitnami/rabbitmq

Password works that is set up using rabbitmq.rabbitmq.password but username is still user even though it shows rabbitmq in the kubectl info

[marcosbc]

as I mentioned before in my case secret contains nonvalid credentials.

@unitto1 But have you checked if that is because of the load-definition property? Note that setting that implies hardcoding the credentials for the RabbitMQ instance, so it does not matter if you set the rabbitmq.auth.user/password values or not. Installing the chart without that option results in a valid deployment, and working credentials.

If you are not using that property, it could be you are referring to a re-creation of the deployment (i.e. via helm upgrade) or re-install, where the chart will re-create the secrets and the correct values will be lost. That is why we recommend obtaining the credentials right after deploying.

Password works that is set up using rabbitmq.rabbitmq.password but username is still user even though it shows rabbitmq in the kubectl info

@pavars Unfortunately I could not reproduce this. I deployed the same command than you, opened the management port and could login as rabbitmq/rabbitmq without any issues.

Could you check if it could be possible that you have a previous PVC with persisted data, that is causing this issue to happen? Check with kubectl get pvc. When you deploy it, the logs of the RabbitMQ pod would show something like this:

 16:34:19.12 INFO  ==> Initializing RabbitMQ...
 16:34:19.15 INFO  ==> Persisted data detected. Restoring...
 16:34:19.15 INFO  ==> ** RabbitMQ setup finished! **

If you do not see anything like that, please share more information (such as the pod stats -i.e. #of restarts- and container logs).

[jacque006]

We just went through an "upgrade" (delete old install, install new version) of our rabbitmq chart from an old version (4.11.1 -> 7.6.8) and encountered this issue. It turned out to be caused by the existing PVCs as @marcosbc mentioned above. When we deleted the install, deleted the PVCs, and reinstalled the chart the auth credentials worked as expected.

I imagine this is not feasible for everyone due to the data loss inherent in that workaround. We store our exchange/queue/etc. configuration in our service code which validates/updates those at service startup. We also use transient messages so the loss of some of that data was fine. And of course you can stop publishing messages and let your queues/messages drain to mitigate some of that.

I would be curious what specifically on the PVCs could cause this issue in case we encounter it again. We used the same user/password/erlangcookie values between the installs and I did not see any Couldn't change password for user 'user'. errors in the logs.

[marcosbc]

@jacque006 Note that in version 7 of the chart it was completely refactored by #2843. So in normal cases that would not have happened, but in this case upgrades from previous versions (< 7.x) to 7.x would not be supported anymore.

Note that those changes were performed with the intention of improving the chart functionality, implementing standarizations present in other Bitnami charts and deprecating a lot of stuff. We do not expect to make such a huge list of changes for a foreseeable amount of time, so in principle upgrades should be easier from now on, from one major chart version to another.

[pavars]

@marcosbc Thanks, yeah I had the issue because of PVC from previous installation of rabbitmq-ha and also testing this rabbitmq helm chart, deleted it and now everything is working as expected

[unitto1]

as I mentioned before in my case secret contains nonvalid credentials.

@unitto1 But have you checked if that is because of the load-definition property? Note that setting that implies hardcoding the credentials for the RabbitMQ instance, so it does not matter if you set the rabbitmq.auth.user/password values or not. Installing the chart without that option results in a valid deployment, and working credentials.

If you are not using that property, it could be you are referring to a re-creation of the deployment (i.e. via helm upgrade) or re-install, where the chart will re-create the secrets and the correct values will be lost. That is why we recommend obtaining the credentials right after deploying.

Password works that is set up using rabbitmq.rabbitmq.password but username is still user even though it shows rabbitmq in the kubectl info

@pavars Unfortunately I could not reproduce this. I deployed the same command than you, opened the management port and could login as rabbitmq/rabbitmq without any issues.

Could you check if it could be possible that you have a previous PVC with persisted data, that is causing this issue to happen? Check with kubectl get pvc. When you deploy it, the logs of the RabbitMQ pod would show something like this:

 16:34:19.12 INFO  ==> Initializing RabbitMQ...
 16:34:19.15 INFO  ==> Persisted data detected. Restoring...
 16:34:19.15 INFO  ==> ** RabbitMQ setup finished! **

If you do not see anything like that, please share more information (such as the pod stats -i.e. #of restarts- and container logs).

no, because I need to setup HA all policies so I actually need that, you don't provide another way to setup pre-defined policies.

[marcosbc]

In this case, the only supported way to add users is like @hansehe did, and is by hardcoding the list of users. This is a limitation of the current container image.

Note that the source code of the initialization scripts are open source, so feel free to take a look and try to improve the current logic, we'd be happy to review new features that improve Bitnami container images: https://github.com/bitnami/bitnami-docker-rabbitmq/tree/master/3.8/debian-10/rootfs/opt/bitnami/scripts

[thomas-riccardi]

We got hit by this when updating from 5.x to 7.x too.
I traced it back to the 7.0.0 breaking change (https://github.com/bitnami/charts/pull/2843): the sts persistent volume mount path has changed from the point of view of rabbitmq:

• < 7: data mounted on /opt/bitnami/rabbitmq/var/lib/rabbitmq, contains config/, .erlang.cookie, mnesia/, schema/
• >= 7: data mounted on /bitnami/rabbitmq/mnesia, contains directly content of mnesia/.

Then, the init script is here: https://github.com/bitnami/bitnami-docker-rabbitmq/blob/a690b5cbc183277a5bf34e4271aee4f1057fd153/3.8/debian-10/rootfs/opt/bitnami/scripts/librabbitmq.sh#L520-L534

The $RABBITMQ_DATA_DIR is not empty as it contains config/, .erlang.cookie, mnesia/, schema/, so it skips the else where the password is reset.
Alas:

• the "Persisted data detected. Restoring..." message misleads into thinking migration is supported
• but in fact no data migration is done
• the password is not reset, so it's the default_password from the .Values.configuration that's used: CHANGEME; explaining why admin credentials don't work.

The changelog only says Backwards compatibility is not guaranteed., which is an understatement: it seems data persistence is completely broken with this upgrade.

The fix is relatively simple though: just move ${RABBITMQ_DATA_DIR}/mnesia/* into ${RABBITMQ_DATA_DIR} if ${RABBITMQ_DATA_DIR}/mnesia/ exists and is not empty
(we should also cleanup the other files/dir: config/, .erlang.cookie, mnesia/, schema/)

That being said, I'm not sure the bitnami docker image is the correct place to do such migration: it's only because of the chart changing the semantic of the volume by mounting it at another level.. maybe a migration script could be done with a helm pre update hook?
In any case I did it manually with that pod:

apiVersion: v1
kind: Pod
metadata:
  name: xxxx-0-migrate-volume
spec:
  restartPolicy: Never
  containers:
  - name: migrate
    image: docker.io/bitnami/rabbitmq:3.8.9-debian-10-r0
    command: ["/bin/bash", "-c", "cd /data; rm -rfv rabbit*; cp -av mnesia/rabbit* .; echo finished"]
    volumeMounts:
    - mountPath: /data
      name: data
    securityContext:
      allowPrivilegeEscalation: false
      runAsUser: 0
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-xxxx-0

(because somewhere in the upgrade where changes in sts spec which are not allowed anyway, I had to delete and recreate the chart anyway, similarly as for redis 7.0.0 (https://github.com/bitnami/charts/tree/master/bitnami/redis#to-700), not sure a helm hook could help there...)

Maybe the changelog could be more explicit, it really helps planning the upgrade. (for example the redis 7 changelog explains the issue with sts changes, that's great.)

[dani8art]

hi @thomas-riccardi, thank you so much for your feedback it seem like your workaround works fine, it is very cool! would you mind to contribute some additional steps to the upgrade instructions? you can say how the paths have changed and you point to your comment as an example. We would be happy to handle the PR if you have a chance to create it.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.