Moby: Docker service update --image "could not accessed on a registry to record its digest"

[sylvainmouquet]

Hello,

I use docker swarm with

docker -v
Docker version 17.06.0-ce, build 02c1d87

My swarm is composed only of one node :
docker node ls 2c9zrmt995p3kxn0qi4sg2xyu * ${nodeName} Ready Active Leader

When i run docker service update --image i have this error :

docker service update --image=${REGISTRY}:5000/${IMAGE}:latest ${serviceName}
image ${REGISTRY}:5000/${IMAGE}:latest could not be accessed on a registry to record
its digest. Each node will access ${REGISTRY}:5000/${IMAGE}:latest independently,
possibly leading to different nodes running different
versions of the image.
{serviceName}
Since --detach=false was not specified, tasks will be updated in the background.
In a future release, --detach=false will become the default. 

The file /home/${USER}/.docker/config.json exists and contains the auth for the container.
When i run docker images, i see docker has been able to download images automatically for others services.
I think the difference is for this service i have downloaded manually the image and after i have create the service.

[sylvainmouquet]

In the same style, docker service has downloaded an image of 5 days ago but there is a new image every day in the registry.
${REGISTRY}:5000/${IMAGENAME} <none> c4dad25bfd4a 5 days ago 1.2GB
If i run docker pull i download the last version

[thaJeztah]

When updating services that need credentials to pull the image, you need to pass --with-registry-auth. Images pulled for a service take a different path than a regular docker pull, because the actual pull is performed on each node in the swarm where an instance is deployed. To pull an image, the swarm cluster needs to have the credentials stored (so that the credentials can be passed on to the node it performs the pull on).

Even though the "node" in this case is your _local_ node, swarm takes the same approach (otherwise it would only be able to pull the image on the local node, but not on any of the other nodes).

Setting the --with-registry-auth option passes your locally stored credentials to the daemon, and stores them in the raft store. After that, the image digest is resolved (using those credentials), and the image is pulled on the node that the task is scheduled on (again, using the credentials that were stored in the service).

Can you confirm if passing --with-registry-auth makes the problem go away?

[sylvainmouquet]

Ok i have not set --with-registry-auth in the command line but why the image downloaded by the service is not the last version ?

[thaJeztah]

but why the image downloaded by the service is not the last version ?

If the service failed to connect to the registry, it didn't resolve the latest digest, so uses any image with the specified image:tag that was already present on the host. Based on your description, docker service update did not pull a new image at all.

Also be aware that when updating the image without changing image:tag, the service may not _update_ the definition (i.e., re-resolve the image), unless --force is set (but I'd have to double check).

[sylvainmouquet]

Yes it's true i have already gone this scenarios but this time i have installed our application on a new virtual machine. We have a script which is in charge to deploy the application. It runs 4 docker stacks with the command line docker stack deploy --with-registry-auth.... I use this script on another environnement which deploy every night our application and for this environnement i have one manager and 2 workers. The manager does not host containers.
With this same script i have deployed the same application but this time the images was not the latest.
Docker service has downloaded images but not the latest version. I am sure that the image has
been downloaded by docker service because the version is defined to <none>.

I use docker stack deploy and the option --force is not present to this command. https://docs.docker.com/engine/reference/commandline/stack_deploy/
Do you have a solution to force the downloading of the latest version every time with docker stack deploy?

[thaJeztah]

For docker stack deploy, there's a --resolve-image=always option, but IIRC, that's the default, so it should resolve the latest image (given that it has the right credentials);

With this same script i have deployed the same application but this time the images was not the latest. Docker service has downloaded images but not the latest version.

What user does the script run as, and is that use logged in? If the script runs as a different user, it will look in _that_ user's home directory for the credentials

[sylvainmouquet]

I use the same user.

[sylvainmouquet]

same bug here https://github.com/moby/moby/issues/34299

[DBLaci]

No #34299 is not the same. The problem I write there is not because of inaccessibility of private registry images.

[sylvainmouquet]

I close this post i will open a new with more information if the problem occurs agin

[majstrooker]

I still get the same error, i.e.:

image dockersamples/visualiser:stable could not be accessed on a registry to record
its digest. Each node will access dockersamples/visualiser:stable independently,
possibly leading to different nodes running different
versions of the image.

Even when using --resolve-image=always and --with-registry-auth. I'm literally following the instructions from https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy. Everything was working perfectly before. Does someone have any other idea on how to solve this?

[XLMBuildManager]

@majstrooker I'm having the same issue.

[trajano]

@majstrooker @XLMBuildManager does it still deploy correctly even with that message? What I found was that it may be a timing issue with Sonatype nexus. I wonder if I can speed it up.

[trajano]

Actually something is really borky with Sonatype Nexus and docker there, I am not sure which part itself is broken, but I didn't have any issues with registry:2

[gurleensethi]

@majstrooker I am using github registry and facing the same issue. Initial Deployment is successful but anything after that I get the error:

image <image_url> could not be accessed on a registry to record its digest. 

Each node will access <image_url> independently, possibly leading to different nodes running different versions of the image.

Did you find any solution?

[thaJeztah]

@gurleensethi that's a known issue with GitHub's registry. The GitHub image registry currently isn't compliant with the registry specification and doesn't implement all options, see https://github.com/containerd/containerd/issues/3291#issuecomment-579804848

[clarkbw]

that's a known issue with GitHub's registry. The GitHub image registry currently isn't compliant with the registry specification and doesn't implement all options

FYI, for those that find this via search. The GitHub Container Registry https://ghcr.io/ is compliant with the spec and this should work correctly when connecting to ghcr.io but won't work when trying with docker.pkg.github.com.