Origin: [oc cluster] Add automatic image pulling

[tnozicka]

Following problems and discussions in https://github.com/openshift/origin/issues/9242 I would like to propose adding option --no-pull=[false] to oc cluster up command. The default behaviour would be to update all origin images when running oc cluster up.

It can happen that you have some origin images from different dates and this can cause problem with compatibility between e. g. registry image and other components.

There is roughly about 10 images you need for running origin from containers (depending on if you are running metrics, ...) and to pull them manually is definitely not user friendly. This will also keep latest images up to date. Since someone may not want them to be updated on purpose we need the --no-pull flag for that.

@csrwng

[csrwng]

When no --version flag is specified, you want one of these things to happen:

1. Use the last released version that corresponds to the oc client version - (as in openshift start)
2. Pull the latest image of openshift/origin from Dockerhub and enable 'force pull' for all images used by the cluster. One drawback is that there's a potential performance hit as the daemon tries to pull every single image it runs.
3. Use the images already present on your Docker daemon (least predictable).

@bparees had suggested that we use (1) as the default, and use a flag for either (2) or (3). @tnozicka seems to favor (2) as the default.

@smarterclayton any thoughts?

[smarterclayton]

I'm not in favor of force pull as a general rule.

1. Is consistent with openshift start.

[bparees]

i'd be fine w/ limiting the effect of "force-pull" to the infrastructure images, not buildconfigs+pods.

[tnozicka]

I wasn't actually proposing "force-pull" (alwaysPull) as a cluster policy. Although it might be useful for a different use case.

I will try to restate:

• When I run oc cluster up ... I want it to run something like:

docker pull openshift/origin:${version}
docker pull openshift/origin-pod:${version}
docker pull openshift/origin-haproxy-router:${version}
docker pull openshift/origin-docker-registry:${version}
docker pull openshift/origin-deployer:${version}
docker pull openshift/origin-metrics-heapster:${version}
...

• Once and at the start.
  
  
  
  • (Possibly check if those images exist and pull them only if they are present. If not they will be pulled when needed. Although this might give you a bigger chance to catch the race condition between pulling images when they are being updated on Docker Hub to end up with different builds. But at least with this auto pulling you may fix it as easily as oc cluster down && oc cluster up.)
  
  

Because if those images are in different versions they might not work together. This happens especially with using version latest because the tag is being overwritten on Docker Hub and you may end up with different versions of those images from different builds.

• If user removes only one of the origin images it will be automatically pulled and he will end up with all images tagged latest but one of them will be old e. g. 10 hours and all the others e. g. 20 days. Or he will try to remove all those images and forgets one and ends up the other way around.

I am proposing for this pull to be the default behavior because it will simply allow user to run oc cluster up and be (almost) sure that all of those images are from the same build. (And also the option to disable it if user chooses to do so.)

[bparees]

I am proposing for this pull to be the default behavior

with @csrwng's PR that will be the default behavior (use the images that match the oc binary version).

[csrwng]

i'd be fine w/ limiting the effect of "force-pull" to the infrastructure images, not buildconfigs+pods.

The current force pull plugin doesn't allow us that level of granularity.

[csrwng]

@tnozicka so I considered doing a pull as you describe, but there's a few reasons that I don't think it's a good idea, especially if we make it the default behavior:

1. One of the nice things about 'cluster up' is that stands up a cluster as quickly as possible. Right now we can accomplish this by pulling a single image. If we force it to pull several images before starting, the experience might not be as nice.
2. Which images do we pre-pull? Each use of cluster up might need a different set of infrastructure images. An exhaustive list may not be a great idea (not including the metrics ones):
   https://github.com/openshift/origin/blob/master/hack/build-images.sh#L82-L123
3. As you mention, we may still end up with inconsistent versions, since you may be pulling as new images are being tagged.

The default of using the latest tagged release seemed like a more sane default. But if you _really_ want to pull the latest, I thought the forcepull would be a good compromise since you could start with it, at least run it once and make sure you got a pull of whatever you happen to use.

[tnozicka]

I am proposing for this pull to be the default behavior

with @csrwng's PR that will be the default behavior (use the images that match the oc binary version).

@bparees Is it? The default value for that flag seems to be false https://github.com/csrwng/origin/blob/aa427ed87ef94875a758e1d70d566572dfaf0620/pkg/bootstrap/docker/up.go#L135

[bparees]

you don't need force pull to get the matching image versions... the images are tagged with the commit id, so if you don't have that commitid pulled, it will be pulled. if you do have it, you don't need to pull them.

[tnozicka]

@bparees I am not sure where this commitId tagging occurs. Would you mind pointing me to that?

I don't see any tags with commitId on Docker Hub https://hub.docker.com/r/openshift/origin/tags/ ; https://hub.docker.com/r/openshift/origin-docker-registry/tags/

[bparees]

@tnozicka sorry, release tag, not commit tag. but same point. your images will match the release of your oc binary.

[tnozicka]

@bparees Yes, but this does not work with latest

[bparees]

if you're building your own oc binary, we can't guarantee the images match your binary anyway because we have no idea what you built it from.

and that is not the primary use case of oc cluster up.

the default behavior will be to match the oc binary to the corresponding released images, and if you don't have those images, they will be pulled (standard docker behavior).

[csrwng]

Even if you build your own oc binary (from master), release tag should match the last released set of images.

@tnozicka does this default (last released version) not work for your use case? I would expect it to give you the most consistent user experience.

[tnozicka]

@csrwng I am building my oc binary from source (make build). The default version there is latest.

@bparees: if you're building your own oc binary, we can't guarantee the images match your binary anyway because we have no idea what you built it from.

That's ok for me. This issue I have is not about incompatibility between oc binary and the server (which is kind of on my responsibility), but between different components of the cluster itself.

I just want to have cluster running from one build of origin:latest. Waiting for a release is not acceptable for most of my use cases. And that's why the latest tags are there, to allow early integration. I would just like oc cluster up to work better with this use case.

I am mostly ok with @csrwng solution, but I am concerned a bit about changing the default policy for whole cluster. (Although I could use that piece of functionality for a different purposes. And since it's not enabled by default, I support that going in. https://github.com/openshift/origin/pull/9865)

@csrwng

1.One of the nice things about 'cluster up' is that stands up a cluster as quickly as possible. Right now we can accomplish this by pulling a single image. If we force it to pull several images before starting, the experience might not be as nice.

I understand your concerns, but if we only pull those images that are already present we would be downloading only small image diff or it would be basically noop (+round trip). Assuming this is done in parallel and probability of having different images is small, I don't see almost any slowdown there.

All this behaviour should be limited to re-writable tags (right now probably just latest).

2.Which images do we pre-pull? Each use of cluster up might need a different set of infrastructure images. An exhaustive list may not be a great idea (not including the metrics ones):
https://github.com/openshift/origin/blob/master/hack/build-images.sh#L82-L123

Yes, I believe it is not easy to know which images are needed for current flags (like --metrics), but it is still doable.

3.As you mention, we may still end up with inconsistent versions, since you may be pulling as new images are being tagged.

Yes, but only if you hit the race condition. The other possibilities will be eliminated by this. Also this time it will be easily solvable by running oc cluster down && oc cluster up. Also the _alwaysPull_ solution is prone to breaking up after pulling one infrastructure image from new build while others stay the _older_ latest. (But still easily solvable the same way.)

The default of using the latest tagged release seemed like a more sane default. But if you really want to pull the latest, I thought the forcepull would be a good compromise since you could start with it, at least run it once and make sure you got a pull of whatever you happen to use.

I need to be running latest images. But you have a good point on running it once with --always as a workaround!

[csrwng]

@coreydaley I think what we want to have is a way to pull the images related to an origin release with a given version. Maybe it should be a separate command from 'cluster up'.

[csrwng]

So the 2 use cases we would like to cover are:
1) you want to ensure that you have the latest images when using --version=latest. Right now we don't re-pull latest if present, so --version=latest might mean you're using last month's images.
2) you are preparing an off-line demo and want to ensure that you have the openshift images you need to run a cluster.

@stevekuznetsov @smarterclayton fyi
Would you be ok with having a separate command from 'oc cluster up' to handle these use cases? Something like 'oc cluster pull-images' ?

[smarterclayton]

Why not just create a bash script someone can run? The offline case is
just not that important. Check a bash script in somewhere and tell people
to run it (take the version as the tag parameter).

I could maybe buy "latest" for the base image pulling the latest.

On Wed, Nov 16, 2016 at 10:27 AM, Cesar Wong [email protected]
wrote:

So the 2 use cases we would like to cover are:
1) you want to ensure that you have the latest images when using
--version=latest. Right now we don't re-pull latest if present, so
--version=latest might mean you're using last month's images.
2) you are preparing an off-line demo and want to ensure that you have the
openshift images you need to run a cluster.

@stevekuznetsov https://github.com/stevekuznetsov @smarterclayton
https://github.com/smarterclayton fyi
Would you be ok with having a separate command from 'oc cluster up' to
handle these use cases? Something like 'oc cluster pull-images' ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/9848#issuecomment-261029322,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pyZV2xORIw_VKXlimh2MIT5x-s6cks5q-0r1gaJpZM4JMawS
.

[bparees]

you want to ensure that you have the latest images when using --version=latest. Right now we don't re-pull latest if present, so --version=latest might mean you're using last month's images.

which is a behavior we absolutely need to preserve. If we want to add a "force pull infra" flag i'm ok with it, but --version=latest behavior should stay as is. That flag is saying "use images with tag FOO", if FOO exists locally, it should be used, not pulled (absent some other flag indicating a pull always behavior for infrastructure images)

[csrwng]

@bparees @smarterclayton so would it be ok to add a --latest flag? Or would you just prefer to have the script checked-in to the hack directory and call it done?

[bparees]

"--latest" feels confusing since i'm going to use it with "--version=latest", and "--latest-images" doesn't actually use the latest images in openshift start.

so i think i'd prefer a totally new name for the flag that doesn't muddy the waters with those other flags/behaviors.

personally i would prefer the flag over a shell script.

[smarterclayton]

The analogue to this request is a flag that does "yum update" on this box
and all boxes in the cluster. That's something we wouldn't do there - why
would we do it here?

On Nov 17, 2016, at 9:03 AM, Ben Parees [email protected] wrote:

"--latest" feels confusing since i'm going to use it with
"--version=latest", and "--latest-images" doesn't actually use the latest
images in openshift start.

so i think i'd prefer a totally new name for the flag that doesn't muddy
the waters with those other flags/behaviors.

personally i would prefer the flag over a shell script.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/9848#issuecomment-261253920,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pwTtJKKS-cP9wASpC1dhqYuKfhYYks5q_F6ngaJpZM4JMawS
.

[bparees]

because if i yum update to the same package level on all my nodes manually, i can expect them to all behave the same way.

if i update to origin 1.4.0 on all my nodes, they still won't all behave the same way if they have different local images.

[coreydaley]

Have we made a decision on this issue yet?

[coreydaley]

@bparees @smarterclayton @csrwng
Has there been a decision made on this issue yet?

[bparees]

my preference is still a new flag that indicates "force pull the infra images", as discussed and justified here:
https://github.com/openshift/origin/issues/9848#issuecomment-261250125
https://github.com/openshift/origin/issues/9848#issuecomment-261253920

[bparees]

At this point i'm with @smarterclayton, i'd prefer to close this as won't fix. The case of "i want to specify --version=latest and i want everything pulled because i might not have a full set of local images but i might have some of them and they won't work with other stuff that's going to get pulled from dockerhub" doesn't really justify effort here.

1) if you specify a "real" version and don't have it, we already pull the images
2) if you specify no version, we figure out the right version (based on the binary) and pull it if you don't have it
3) if you specify --version=latest you probably want to use the local images you just built

the case (4) of "you specify --version=latest and you want to pull whatever is latest for all the images" seems unlikely given the ability to do 1+2.

[coreydaley]

Closing as "won't fix" as per @smarterclayton and @bparees