Moby: Are we forgetting `docker secret update` command in cli?

[realityone]

Hi there,

I'm trying to use docker secret to manage our private keys, but I find that we support /secrets/<id>/update in API, but no docker secret update command in cli, even in client.

• https://github.com/docker/docker/blob/master/client/interface.go#L164
• https://github.com/docker/docker/blob/master/api/server/router/swarm/cluster.go#L50

Are we just forgetting it or it's by designed?

I'm working on Docker 1.13.0-rc4.

[realityone]

Oh, this is by designed at https://github.com/docker/docker/blob/master/daemon/cluster/secrets.go#L109.

[thaJeztah]

Updating is currently an API only feature; the secret (data) itself is immutable, and only "labels" can be updated. We decided that having a secret update command would be more confusing than "not" having this, so (at least for now) updating through the CLI is not implemented.

Thanks for checking though, and paying attention :smile: 👍 !

[s4s0l]

Hmm, so in order to update secret the only available procedure is to remove all services using it, recreate secret and start services from scratch?

[thaJeztah]

You cannot update the secret itself, but you can update the service that's using it, and add a different secret;

echo foo | docker secret create secret-1 -

docker service create \
  --secret src=secret-1,target=db-password \
  --name web \
  nginx:alpine

echo foo2 | docker secret create secret-2 -

docker service update \
  --secret-rm secret-1 \
  --secret-add src=secret-2,target=db-password \
  web

[s4s0l]

Reasonable as not many images will support password change on the fly. Thanks.

[thaJeztah]

One of the considerations to make the secret itself immutable is to allow rolling back a service definition to a previous version. A PoC version of the feature had built-in versioning of the secret itself, but this added a lot of complexity with not a lot to gain, so if was decided to keep this out.

[kleptog]

TBH, when rolling back to a previous service definition it doesn't seem at all unreasonable to get the new version of the secret. After all, the secret (generally) gives access to a resource outside the swarm, and that is not being rolled back.

Not having an update command is also annoying when you are deploying your infrastructure via ansible. It means that password rollover will never happen because you can't reliably script everything that needs to be updated.

Interestingly, "docker stack" does have all the necessary info to rebuild the system, but it would be weird if docker stack changed the name of the secret for you.

This is one of those situations where I wish docker would give you just a bit more rope.

[RehanSaeed]

You cannot update the secret itself, but you can update the service that's using it, and add a different secret;

That seems suboptimal. I expected to see an update command too. It would be nice if the secret was updated in running containers too but I'd settle for having to redploy a service if I could get my hands on this feature.

Can we consider re-opening this issue?

[vizv]

And it's quite annoying change compose file every time when updating a config or secret when using docker stack.

[mlescaudron]

I don't understand why the command is still unavailable because of UpdatedAt field :

[
    {
        "ID": "l2qnmoxtlopsgij9ayu4pgbsj",
        "Version": {
            "Index": 47914
        },
        "CreatedAt": "2017-09-20T03:46:16.035020938Z",
        "UpdatedAt": "2017-09-20T03:46:16.035020938Z",
        "Spec": {
            "Name": "test_1",
            "Labels": {}
        }
    }
]

So, while waiting, I created a script that allows to do it easily.
It updates all services using the secret.
If this interests you : https://gist.github.com/MLescaudron/e8248d32d3a5b8caaf622c1a829cf067

[rgardam]

Also this is quite painful as you cant use variables in the docker-compose file for secret versions.

[stevelacy]

As a workaround for using docker stack deploy (which would need to be modified for the new secret name) I am using this script to update a swarm wide secret.
It does have the issue of two docker service update rolls.

https://github.com/stevelacy/docker-secret-update

[ilyasotkov]

Follow this issue https://github.com/docker/cli/issues/667 and pull request https://github.com/docker/cli/pull/668, which will add a "name" field to both configs and secrets. It will allow updates via environmental variables like so:

version: "3.5" # future Compose file version

secrets:
  my-secret:
    name: my-secret-${VERSION} # name of the Docker Secret with variable interpolation
    file: ./secret.html

configs:
  my-config:
    name: my-config-${VERSION} # same thing for configs
    file: ./default.conf

services:
  web:
    image: nginx:1.13.6-alpine
    ports:
      - "80:80"
    secrets:
      - source: my-secret # reference by the name of the key, consistent with how it is for volumes now in 3.4
        target: /usr/share/nginx/html/index.html
    configs:
      - source: my-config # same thing for configs
        target: /etc/nginx/conf.d/default.conf

[yangm97]

So, the solution is adding arbitrary names to secrets? Why not do it like environment variables management? How do I remove orphaned secrets?

[immerrr]

Updating is currently an API only feature; the secret (data) itself is immutable, and only "labels" can be updated.

Ok, but the problem is that the labels cannot be updated without hipping through the hoops of finding all services that use that specific secret label, unlinking them manually and then doing the rm; create.

Making secrets immutable is a nice feature, so why don't do it the Git way? I mean, you already do it with images and their hashes, where each image is uniquely identified with a hash, but you can also add a human-readable tag and then it will be resolved upon container creation. Then you are free to push a new image and update the tag, but the old image is still there referenceable by its hash until it's garbage-collected. @thaJeztah, was there a reason why the same approach was not applicable* for secrets?

[thaJeztah]

@immerrr see some of my comments in https://github.com/moby/moby/issues/35048#issuecomment-333778750 (with a possible approach)

[dlibanori]

It is very frustrating to manage secrets this way :(

I would love to have my secrets versioned, just update it and docker relaunches my services, just rollback it and docker gets my previous secret.

[timrazik]

I know this is an older thread but as someone who manages a large number of secrets, I have a suggestion here. If you want to change a secret with minimal to no impact you can create credentials that work in parallel. This comes in handy if you want to expire credentials periodically to meet security compliance or close gaps in cases where people change teams or leave your organization. Let's take an example of a database ID and Password. What we can do is actually create TWO sets of credentials with the same security permissions.

database:  myDockerDB
userId_A:    appUserID_A
password_A:  some_password#1234
userId_B:  appUserID_B     // This User would be INACTIVE for now.
password_B:  some_other_password#9876

In our docker stack/compose file we would then map our secrets as such

services:
  myService :
    image: dtr.docker.com/mycompany/dockerService:1.0.0
    environment:
      - JAVA_OPTS=-Xms512m -Xmx512m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:-TieredCompilation -XX:+ParallelRefProcEnabled
    deploy:
      mode: global
    networks:
      - bridge
    configs:
    secrets:
      - mode: 444
        source: userId_A
        target: userId
      - mode: 444
        source: password_A
        target: password

Now here comes the fun part. When we want to change the password, what you do is have your DBA ACTIVATE userId_B and you set your password for it. BOTH userId_A and userId_B would be active for an overlap (say 7 days).

You would then just update your stack file or service update to map the B credentials. This would be allowed because no service is utilizing B. The beauty here is that no code change is required as you are not changing the key that the code is using to look up the credentials. Now you can roll through your service(s) during the overlap period. Once complete you have your administrator deactivate the old account so that it can no longer be used.

services:
  myService :
    image: dtr.docker.com/mycompany/dockerService:1.0.0
    environment:
      - JAVA_OPTS=-Xms512m -Xmx512m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:-TieredCompilation -XX:+ParallelRefProcEnabled
    deploy:
      mode: global
    networks:
      - bridge
    configs:
    secrets:
      - mode: 444
        source: userId_B
        target: userId
      - mode: 444
        source: password_B
        target: password

[Sammons]

This issue overall is a massive thorn. I cannot even do --secret-rm and --secret-add at the same time to replace a secret if they have the same target - I get an error about a conflicting target.

[thaJeztah]

@Sammons what version of Docker are you running? I tried to reproduce that situation, but wasn't able to;

$ printf "secret1" | docker secret create secret1 -
$ printf "secret2" | docker secret create secret2 -

Create a service using secret1

$ docker service create --secret source=secret1,target=/somewhere --name myservice nginx:alpine
7mamd13lh282p5d5i423lmabd
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service converged 

Update the service, and replace secret1 with secret2

$ docker service update --secret-rm secret1 --secret-add source=secret2,target=/somewhere myservice
myservice
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service converged 

If you have ways to reproduce on a current version of Docker, could you open an issue for that?

[Sammons]

@thaJeztah I will try updating to 18.* and get back to you. I know these nodes are running 17-something.

[thaJeztah]

@Sammons it could be a client-only change, so for a quick workaround, you could just try with a more recent version of the Docker CLI

[Sammons]

@thaJeztah for reference this situation happened with client version: 18.05.0-ce and with server version 17.12.0-ce; I am trying to update the server to fix.

Update: still persists with server version 18.03.1-ce - I will try to put together an easy reproduction

[Sammons]

@thaJeztah Got it. The problem arises when I try removing a secret that doesn't exist instead of the correct secret, so the error is correct. The error thrown is just not pointing to my root problem (that I'm removing a secret that does not exist). 👍

[thaJeztah]

Ah; looks like something to improve: could you open a new issue for that with minimal reproduction steps?

On 24 May 2018, at 03:55, Ben Sammons [e-mail] wrote:

@thaJeztah Got it. The problem arises when I try removing a secret that doesn't exist, so the error is correct. The error thrown is just not pointing to my root problem (that I'm removing a secret that does not exist).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[markvr]

+1 for implementing a simpler way to update a secret please! Currently we store compose files in version control, with encrypted passwords as environment variables, and a simple script on Jenkins that decrypts them before deploying. This means the passwords are in plaintext in the services.

We'd like to remove this and move to secrets but the whole theatre around secrets is just too complicated. Trying to maintain secrets in a CI pipeline requires a lot of custom work to encrypt them in version control, generate a unique name for the secret, deploy the secret before the service, then update the compose file to have the new unique name for the secret (or keep track of an external variable name), deploy the compose file, and finally remove the old secret. What a faff!

My ideal situation would be a public/private key pair, which is used to encrypt the secret. The compose file then looks like:

secrets:
   my-secret: ENC(......)

This can safely be version controlled, and when deployed Docker then decrypts the secret with the private key, and updates any services that were using it. This means the encrypted value of the secret is treated the same as any other configuration value, and would massively simplify getting secrets out of environment variables.

I realise a lot of people probably externalise their secret management using something like Hashicorp Vault or similar, but that is a whole extra product, and it would be really cool if this could just be kept simple within Docker.

@thaJeztah - Could this issue be reopened, or is it worth me creating a new issue with this feature request?

[jamiejackson]

Adding my script to the list: https://gist.github.com/jamiejackson/a1818acedaeb9c3cd70bafac86a0100b

./docker_secret_update.sh --stack=hudx_local2 --service=lucee --secret=fr_admin_password --value=foobar

As far as I can tell, it might be an improvement over @stevelacy's (which uses another file and needs two service updates). I just finished it, though, so I'm not sure what its flaws will be.

[BluSyn]

I am frustrated by the need for complicated solutions to a simple problem. Using docker-compose + secrets w/ docker stack deploy is essentially impossible with the current paradigm.

It should not be required to manually update the docker-compose file every time I update the secret. Currently not changing the secret name in compose file causes docker stack deploy to fail if any secret file has been changed. This is frustrating since this would not happen in any other context: if anything else in the docker-compose file has changed, even external files such as an env file, docker automatically updates the services as one would expect. The current behavior makes auto-deployments + secret management essentially impossible while still using version controlled configuration.

Sharing the script I am currently using to solve this problem:
https://gist.github.com/BluSyn/71a2040d610e06bf8ee06a9480d17fd3

This script will automatically find any service(s) on the swarm using the secret with the specified file name and update them with a new secret. Should be run on a manager node.

This solution requires direct parsing of both the docker-compose.yml file and the current state of the swarm. The advantage is future runs of docker stack deploy will succeed. It will have a side-effect of re-creating secrets + re-deploying all services manually updated since the last full deployment. One nice thing of this script is the limited information an admin needs to give in order to update a secret and doesn't require a full stack deploy.

In my opinion this is exactly how docker stack deploy should handle secret file updates. This would allow secrets to maintain their "immutability", but the expected behavior of deploying with a compose file is maintained.

[iFreeon]

I'm using docker for the first time and also puzzled why I can't update secrets. I'm deploying stacks using docker-compose.

How do rolling updates work if I have to take the whole stack down to remove and re-add a secret?

[kennell]

I really is silly that this is not supported. Consider a web application where API keys for third-party services are rotated on a regular basis etc. -- tons of use cases where it would be useful to update a secret.

[val-roam]

we store our ssl certs and keys in docker secrets, when its time to update those - its a massive pain in the ass! have to go hunt down everything that uses these certs (which is..everything) change the secret name to _v2 or _v3 or whatever, redeploy, argh ...

and that's just ssl certs. there're plenty of secrets that need to be rotated on regular basis, and much more often than ssl certs!

what kind of work is being done around solving this situation?

[er1z]

Reopen, please!

[DD202]

A pretty straight forward solution is to calculate the hash of the file being used as the secret and inject that as the name of the secret. This way if the secret changes, a new one will be created during the docker stack deploy process. A process is then needed to clean up orphaned secrets long term, but it eliminates a lot of the headache from a dev standpoint.

[jamiejackson]

@DD202 You and I must have different definitions of "straightforward," but that's exactly what I'm doing.

Although I have this going in production, it wasn't quite ready to be untangled and put into a repo for public consumption. However, I've just posted it in its raw and ugly state (and maybe I'll get around to making it more presentable some day).

https://github.com/jamiejackson/docker-secret-hash

[Xanatosinc]

I landed here trying to Google my way through this. This approach could still maybe be improved upon, but there's a one-line command for rotating secrets about halfway down the page here (two lines if you include creating the new secret), under Rotating Docker Secrets: https://semaphoreci.com/community/tutorials/managing-secrets-in-docker-swarm

Suppose you have an existing service service_app with a connected secret secret_1 which is mapped to /path/in/container:

docker secret create secret_2 /updated/file/in/host
docker service update --secret-rm secret_1 --secret-add source=secret_2,target=/path/in/container service_app

[r0039]

In fact, docker secret should be managed like Git way or version control. It is really useful to to manage, scale the applications.

I give an example:
First day at new company, the existing SSL has just expired. So I have to update the docker secret with the new SSL. On docker manager, all of docker containers are using the same docker-secret-name. So better way to backup the current docker-secret-key, then update the existing docker-secret with a new SSL. After that, I only need to re-run docker services to apply the change.

That's save time. Thank you and hope you consider to code this feature.