Swarmkit: Different image selection for "docker run" and "docker service create"

[boriska70]

There is a difference in the behavior of "docker run" and "docker service create" commands. Assuming that I have image stored in Dockerhub and then I build a new one locally:

• "docker run" will use the local image
• "docker service create" will use the remote image (from Dockerhub)

While it eventually makes sense (if service comes as a part of swarm that serves clusters), there are 2 potential issues:

1. This behavior is not mentioned in any place that I checked - worth documenting it
2. It introduces some complexity on testing local changes. There are workarounds (use "docker run", push with a special tag, etc.) but it would be useful to provide a flag for "docker service create" command that enforce it to try using the local image first (if presented)

docker version output:

Client:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 04:02:03 2016
 OS/Arch:      linux/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 04:02:03 2016
 OS/Arch:      linux/amd64
 Experimental: true

[stelund]

I think it's vital that it pulls even though there exists an local image. At least by default.

A work around to pushing with a specific tag would be to run your own registry. Docker distribution comes as an ready to use image and easy to spin up while testing. Choose the tag name by localhost:5000/[:version] or whichever port you like, then push it.

[boriska70]

I understand the importance of pulling image from registry when working with cluster. What I suggest is to document this, because it is not so clear for people who just start working with docker. Thank you.

[n8behavior]

I think the real issue is that we should be consistent about how we resolve tags to SHAs. Here's how I believe tags (even latest when implied) are resolved for these commands--correct me if wrong--though I think my argument stand regardless???

1. docker run -- local repository is tried
2. docker service create -- local repository is NOT tried
3. compose bundle (used later by docker deploy) -- local repo is tried

I think the default behavior should be to resolve the tag to the SHA of the local image if one matches--more on that "matches" part later. I like this because it is the path-of-least-surprise. It ensures that my local image (if present) is what gets used--not, for example, a newer image in the registry added by someone else.

Back to that matching bit...and I think this is the real issue underneath...

Let say you docker run bar and it matches bar@sha123acb... locally but bar@sha456def... on the remote server. It's just like handling source code. You have to know what version you are at or your CI/CD pipeline gets very wonky. In other words, we need to ensure things are repeatable. To that end, I do think it's a good idea to "push" before running/deploying a new container. Local repositories should be consider ephemeral, IMO.

One last thought, in the example above we "implied" latest as our tag, but the problem is the same with any tag. You have no guarantee that your image match the remote's unless you pull the SHA of that tag locally. The same is true for using a private registry or not i.e. dtr.yourdomain.com/bar or yourorg/bar. The fact you or Docker are hosting the remote has no bearing on the issue.

To be clear, resolving tags locally is my preference, but I think the real issue is that we need to be consistent.

[aluzzardi]

Indeed - you can use a digest to pin to a particular image (e.g. image:tag@sha256:digest). We've discussed in the past automatically resolving this to a particular digest on the engine side.

@aaronlehmann @stevvooe: Thoughts?

/cc @sfsmithcha @mgoelzer Maybe we should mention this somewhere

[stevvooe]

The main difference is that service create always tries to update. I am not sure what "local repository is NOT tried" -- this makes absolutely no sense in a clustered environment. Typically, deciding the current value of a particular image needs to be centralized for a distributed service. For swarm mode, we use the registry to make this decision but the client can make the decision by using an image digest.

Pin the image with a tag or digest if you want consistency.

We may do this client-side, automatically, as that will centralize the resolution.