Moby: force docker deploy to pull new images

[raarts]

(running docker 1.13.1 experimental mode, using docker deploy w/composer v3 file).

Occasionally, for reasons unknown to me, updating a stack by running docker deploy does not check for and pull newer images. I don't know how to recover from this other than doing docker pull on each host. docker stack rm xxxxx, and re-deploying does not help.

Specifying the sha would help, but I don't see how that is practical, when deployment is run from a CD pipeline.

I searched for CLI flags or existing issues, but couldn't find any.

When I issue a docker pull <image> the swarm manager I'm talking to reports the image as up-to-date, but if I login to another manager, it does download the newer image.

[vdemeester]

/cc @nishanttotla @aaronlehmann
I don't remember what is the default behavior for docker service command ?

[nishanttotla]

@vdemeester let's say you have a service created using just a tag, such as docker service create img:tag ..., then the digest is resolved and the service is created with img:tag@sha256:digest.

When you run docker service update, then the image won't be updated unless the --image flag is provided. That is, docker service update --image img:tag .... In this case, the digest will be resolved again, and if the image under this tag was updated, then it'll be pulled.

See https://github.com/docker/docker/issues/24066#issuecomment-259846101

[thaJeztah]

The docker service update command has a --force option which forces an update (ultimately leading to resolving the current digest); perhaps docker stack deploy should have this option as well?

Ideally though, it's recommended to use either a fixed tag, or an immutable digest in the docker compose file

[aaronlehmann]

I'm pretty sure docker service update --force will not switch to a new digest. If you want to update to a newer version of the image, just use docker service update --image myimage:tag servicename. If the tag has changed to point to a new version, this will pin the service to the updated digest.

[thaJeztah]

Using --force allows you to update the image without having to update the tag; just checked, and it seems to require including the image yes;

docker service update --force --image foobar:latest myservice

Does "re-resolve" the digest

[santiagopoli]

Are there any plans to allow force updating the latest digest without providing --image ?

[aaronlehmann]

Are there any plans to allow force updating the latest digest without providing --image ?

No plans that I know of. I suppose we could have a flag that caused the digest to be re-resolved based on the tag, but there are already a lot of flags and since this is something that can be done without a dedicated flag, I'm not sure it's justified.

[goetas]

@thaJeztah

perhaps docker stack deploy should have this option as well?

would be really helpful

[NagoLazaro]

@thaJeztah @raarts

The docker service update command has a --force option which forces an update (ultimately leading to resolving the current digest); perhaps docker stack deploy should have this option as well?

From my point of view I think that it will be really quick & easy to deploy a new version of the stack just by having this option.

When working in a Swarm Cluster, it will be quite helpful to "upgrade" the services version calling something like:

docker stack deploy --force -c compose-file.yml STACK
or
docker stack redeploy STACK

Which could gradually pull the images of the new services, and deploy the instances.

By using the healthchek option someone could achieve zero-downtime when "updating" or "upgrading" their services in a Swarm cluster.

Hope to hear something about these soon. Thanks in advance!

[biels]

Totally agree with Nagoo.

A docker stack deploy --force -c compose-file.yml STACK is needed. After all, docker swarm is meant to be declarative. When you update the definition of a stack, you expect the swarm to converge to the desired state.

And you can say "Well, then just bump the tag version numbers".
I feel like that's the right thing to do for production environments, but not for development environments where you will be redeploying (to a local dev environment) constantly, polluting the local image cache with thousands of nearly identical images.

More thanks in advance, in addition to those given by Nagoo!

[dnephin]

This should be fixed in 17.09. The docker stack deploy --resolve-image flag was added, which defaults to always, so a deploy should cause the server to check for update and update the service if the image has changed.

[rdxmb]

Just tried this with

# docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:24:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:23:00 2017
 OS/Arch:      linux/amd64
 Experimental: false

and portainer/portainer:latest (this is very friendly for testing because they have many updates ;)

docker stack deploy --prune --with-registry-auth --resolve-image always -c production-stack.yml 
# portainer 1.15.3 is used
docker pull portainer/portainer
docker stack deploy --prune --with-registry-auth -c production-stack.yml $(basename $(pwd))
# portainer 1.15.3 is used
docker stack deploy --prune --with-registry-auth --resolve-image always -c production-stack.yml $(basename $(pwd))
# portainer 1.15.5 is used

So it seems for that you have to use
1)docker pull AND
2) --resolve-image always

What I expected is that with
--resolve-image string Query the registry to resolve image digest and supported platforms ("always"|"changed"|"never") (default "always")

2) I do not have to use that option, because it is the default
1) docker pull is not neccessary

Or should this only working with a specific tag?
Unfortunately there is no portainer/portainer:1.15 or portainer/portainer:1 . So what is the reccomended way to get the updates with docker stack deploy?

//edited:

so a deploy should cause the server to check for update and update the service if the image has changed.

Has changed where?
On the manager?
On the worker?
Or on the registry?

[rdxmb]

Another test:

# docker version
Client:
 Version:      17.11.0-ce
 API version:  1.34
 Go version:   go1.8.3
 Git commit:   1caf76c
 Built:        Mon Nov 20 18:37:39 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.11.0-ce
 API version:  1.34 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   1caf76c
 Built:        Mon Nov 20 18:36:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Portainer was running version 1.14.x
docker stack deploy -c production-stack.yml $(basename $(pwd))
still 1.14.x . But the Browser-session was killed and I had to login again. (This is what had happened in the example above, too.)
Remove some other services from the stack file, run
docker stack deploy -c production-stack.yml $(basename $(pwd))
updated to 1.15.

[rdxmb]

I guess there was a testing problem at my (and maybe others) site - as I commented here:
https://github.com/moby/moby/issues/31357#issuecomment-367761212

[hholst80]

Besides that the defaults are wrong, imho, this seems to imply a huge performance penalty. Timing this with --resolve-image always vs. --resolve-image changed below. I would much rather have this as an option in the compose file than to have a default that causes such a slowdown. A pull to update should be explicit as it might actually break a working stack if the upstream repository changed.

No changes applied locally:

$ time docker stack deploy -c docker-stack.yml --resolve-image changed rapids-dask
Updating service rapids-dask_worker (id: 9b2cdoqzbzl2rknm1dergmsqc)
Updating service rapids-dask_notebook (id: kook3m3rbj7h36ivpwto4posu)
Updating service rapids-dask_scheduler (id: ik0taknqmm7t41a2kw0rd7cei)

real    0m0.060s
user    0m0.020s
sys     0m0.020s

$ time docker stack deploy -c docker-stack.yml --resolve-image always rapids-dask
Updating service rapids-dask_scheduler (id: ik0taknqmm7t41a2kw0rd7cei)
Updating service rapids-dask_worker (id: 9b2cdoqzbzl2rknm1dergmsqc)
Updating service rapids-dask_notebook (id: kook3m3rbj7h36ivpwto4posu)

real    1m26.647s
user    0m0.016s
sys     0m0.035s

[balthild]

Using --force allows you to update the image without having to update the tag; just checked, and it seems to require including the image yes;

docker service update --force --image foobar:latest myservice

Does "re-resolve" the digest

@thaJeztah No, it does not. After executing docker update with --image foo/bar:latest and --force args, the image version on nodes is still not updated. I can confirm this behavior by showing SHA of the image on node (docker inspect --type image --format '{{index .RepoDigests 0}}' foo/bar:latest).

# docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:43:34 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:13:40 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[thaJeztah]

I can confirm this behavior by showing SHA of the image on node (docker inspect --type image --format '{{index .RepoDigests 0}}' foo/bar:latest).

When deploying a service, the images are pulled, and referenced, by digest, so the pull triggered by the service update won't update the existing foo/bar:latest in the image cache on the node.

To verify which image the service is running, use docker service inspect or do docker container inspect on a container that's part of the service

[balthild]

@thaJeztah docker container inspect shows that the image is foo/bar:latest. Also, If I go into the container with docker exec -it foobar.XXXXX sh on a node, I can check the files to verify that image is not pulled or updated.

[thaJeztah]

@balthild is foo/bar:latest in a private registry? i.e., does it need authentication? If so, it's possible that it's unable to resolve if the authentication stored in the service has expired, in which case it won't be able to resolve and pull (and then falls back to using the local image)

[balthild]

@thaJeztah Yes, it's private. But if I delete the service and remove the images on the worker manually, then re-create the service, the worker CAN pull the latest image. This shows that the issue is not caused by authentication.

Anyway, I've switched to k8s. Docker swarm has wasted my time too much.

[cocowalla]

@ thaJeztah

When deploying a service, the images are pulled, and referenced, by digest, so the pull triggered by the service update won't update the existing foo/bar:latest in the image cache on the node.

Is there some way to update the existing foo/bar:latest in the image cache on the node? It's confusing doing docker images and seeing foo/bar:latest, when a newer image is actually running.

[trajano]

I wrote a script that would do the redeploy locally https://gist.github.com/trajano/82982c229948df445ada518d7c6273df