Compose: docker-compose up doesn't pull down latest image if the image exists locally

[kag0]

It would be nice if there were an option to check for new versions of images when running docker-compose up.

We already have this functionality with docker build --pull as was discussed here https://github.com/docker/docker/issues/4238 and there is an open issue to bring --pull to docker run here https://github.com/docker/docker/issues/13331.

I propose adding --pull to up to always attempt to pull a newer version of the images in the compose file.

[shin-]

Is there a reason why docker-compose pull && docker-compose up is impractical?

[kag0]

I think most of the points for/against are the same as the ones discussed in the issue for adding --pull to docker run. They range from ux and consistency, to ease of scripting/workflow, to swarm integration (out of curiosity, what does docker-compose pull do with swarm?).
I don't think it's a major issue, but something to consider. The same type of users who desired the feature elsewhere would likely also enjoy it here.

[decoomanj]

I'm trying to execute "docker-compose build", but it doesn't refresh the image referenced in the Dockerfile, except when you use _--pull_.

You can also build containers during the start with _up --build_. But the newest images will not be pulled. Can we expect sthg like "docker-compose up --build --pull" (or similar)? Maybe it makes sense to place it in the YML since not all build must be refreshed (cfr. local images).

[cressie176]

Instead of (or in addition to) adding --pull to the cli, what about adding something on the service definition in the docker-compose file?, e.g.

version: '2'

services:

  postgres:
    image: postgres
    container_name: postgres
    pull: true
    ports:
     - '5432:5432'

This way if there's a service I don't care about being latest and one I do, docker-compose won't waste time downloading images I'm not interested in

[blockloop]

I came here looking for this feature because we use it in our production cluster of Kubernetes. There the tag is "imagePullPolicy" and it can be set to "IfNotPresent", "Always", or "Never." Something similar for a compose environment would be nice.

[PengBAI]

In our case, we need to rebuild base image everyday to make sure the latest dependencies updated for applications. Docker compose up to pull the latest image with the same tag is a nice feature. Why not !

[marcelovcpereira]

Any news about this?

[sugun999]

Hi,

Any news about this issue?

[jonathan-kosgei]

+1

[ivytribe]

+1

[shin-]

As I mentioned earlier, docker-compose pull && docker-compose up is functionally identical. There's no good reason to add another flag for it.

[blockloop]

Imagine that git didn't have pull because git fetch && git merge origin/master is functionally identical.

[telly23]

Adding a pull: true tag could be useful for example if some of the images you use in your compose file are in your cache. docker-compose pull pull _all_ of the images in your compose file, and that pull will fail if these images are in your cache but not in the repository.

[fulllove890]

+1

[JustinLivi]

One scenario in which docker-compose pull && docker-compose up becomes impractical is when you're using multiple docker-compose files. You can easily end up with a command like docker-compose -f docker-compose.test.yml pull && docker-compose -f docker-compose.test.yml up.

[xellsys]

We have a scenario in which we develop locally and we would only want to pull some of the images from remote. The one (or more) developed locally should remain untouched. In that case we are obligated to build/pull images manually before running docker-compose up.
A pull: true would be beneficial.

[lig]

@shin- what about rethinking your decision on this? I believe comments and reactions to them look self-descriptive enough.

[shin-]

No, I'm good. This is an open-source project, if you disagree with the conservative approach we take when adding features, you're welcome to fork it.

[alwye]

Agree on adding either a flag to the docker-compose up command or a parameter to the configuration. We use a base image with additional configuration that tends to change frequently during the development process. We want to create a fool-proof environment, where a developer can simply run docker-compose up without debugging what they don't need to debug.

I actually hit this thread after my colleague was checking my pull request and said it broke the build. And the reason being is that the base image was simply missing a couple of packages - but got executed in the final Dockerfile. It would be a good feature, but apparently not something you used, @shin- .

We'd be happy if you introduce this feature in the new version.

[kgalli]

@shin- I think @blockloop showed with the git example how convenient / useful the pull option is. Honestly I hope for such things it is not necessary to fork any project.

I totally understand the conservative approach but it feels like it is not even considered as an option anymore. Maybe it can be part of an upcoming version?

[pwFoo]

A pull: IfNotPresent would be nice. So it could be possible to use fallbacks like
1) use local image
2) pull if isn't local
3) build if not able to pull

[jrich523]

@shin- you keep asking for a reason why the && method wont cut it, and my reason is this. I use it for a "app" image for testing (Puppet PDK/Onceover). The compose file is part of the template repo so when a puppet dev (really ops folks) needs to make a new module they fork that repo. Jenkins runs the image for merge validation on that module repo (internally we have a jenkins plugin that handles the update for the jobs.) Now the folks who use this wont be docker experts, and having to tell them to do the && is an extra step that they can (and probably will) screw up. I dont see why it would be hard or what disadvantage it would create but this reasoning seems like a worthwhile reason to add it. It helps us developers send things out that require less directions and steps..

the short of it is.... to protect against laziness

[ostrolucky]

Here is better reason: && is synchronous. But docker-compose has built in great support for parallel executor which optimizes this stuff. docker-compose up --pull --build could start building image and run it as soon as it's pulled, instead of waiting for all images to be pulled and only then start building

[sgohl]

@shin is assuming one's using for ourselves on a very rare basis with rare updating Dockerfiles.
But if you update a Docker image every day that developers will use, it will rapidly lead to development problems if just one developer forgets to pull the image which he has to do every day (especially those who are not really familiar with docker, and that's actually not their business to know and remember it). Disaster programmed.
Is it a big problem to just add this option to the docker-compose.yml ?
I mean, it will not change other things, it just adds functionality. ..

it's the killer reason why I can't use docker-compose and need to write wrapper scripts with legacy docker run commands , but that's ugly.

[themanifold]

Trying to figure the circumstances under which this ticket was closed - would someone please elaborate? If it's any help, I am pro adding a --pull flag to docker-compose up.

[orodbhen]

I think there are two proposals being discussed here:

1 - Should this be added as a command-line option? Actually the posted issue.
2 - Should this option be added to the YML file.

I definitely agree with the latter, though @shin didn't actually comment on that. To dismiss it with the same argument, that it's functionally identical would be incoherent. Everything in the YML file is functionally identical with the command-line.

I can see his point in regards to the former, but I think the rationale is a little too broad. I can do just about anything on the command-line by chaining together a series of statements with &&. Let's be clear, those are two commands, not one. The criteria should be: Is there enough demand for this that it should be possible to do it in one step instead of two? Because if it's used often enough, the number chained commands just keeps growing. The point is, when you write a script, you want it to be as succinct as possible.

[lig]

@orodbhen no need to argue. There is nobody listening to us here.

[pho3nixf1re]

To add to the reasoning for adding a flag... over the course of the last year or so I have found myself searching for this very thing and have ended up here only to find that I had already thumbed up several comments in favor of a --pull flag. I imagine I will again find myself here the next time as well.

[kag0]

@shin-, please re-open or lock this issue. It's been open for almost two years, receiving constant commentary (both intelligent and entertaining), dozens of participants, and hundreds of votes.
However, it seems clear that the compose team is not interested in pursuing this feature. So let's not waste anyone's time or give room for false hope that the issue will be revived if that's not the case.

[shin-]

Please refrain from using profanities.

[bdharrington7]

While I do think there is usefulness in the original request, a lot of people in this thread seem to forget the spirit of open source: if it’s that important to you, you are free to fork and modify to your heart’s content. I understand that you may not want to maintain a fork, but complaining that the maintainers won’t implement a feature is counter-productive: that’s not how features get implemented, and will make the maintainers want to help you less.

It doesn’t really matter that there’s demand for a feature, especially if no one is paying to use the product. There’s more to consider than just getting every feature that everyone wants into the main product. We should respect @shin-‘s response on this, and believe there are good reasons for not implementing it.

[orodbhen]

@lig Not looking to argue. Just making the case.

Depending on your need, forking may be overkill. In my particular case, I've found Compose to not lend very well to scripting, unless it's a very basic script. Using the Docker Python API and and my own YAML file to preserve settings is more versatile and often simpler.

[er1z]

@bdharrington7 — but you have to maintain your fork and keep it installed on all machines you use (what is rarely possible). Caveat is the Docker Compose is popular, others would be said: „who cares?”

[gregpakes]

The reality is that the comment "it's open source, create your own fork", is just not realistic. The overhead of maintaining your own fork and keeping it up to date with the latest changes from the main repository makes it rarely worth the investment. A much better approach is to petition the maintainers and provide proper reasons as to why the feature is important.

Sadly that will always result in issues like this one with some dissatisfaction. I think the main issue here is that there doesn't seem to have been a proper case put forward as to why this is a bad idea. The argument

"We should respect @shin-‘s response on this, and believe there are good reasons for not implementing it."

just isn't going to satisfy people. The community need to see the "good reasons".

[bdharrington7]

The point remains that pounding your fists and begging will rarely result
in getting what you want, a lot of which was happening in this thread.
There have been plenty of good use cases brought up in the thread as well,
but unless you’re actually paying for the software, there’s no obligation
from maintainers to do anything about it, nor give any reasons why. I’m
just giving @shin- and company the benefit of the doubt here.
On Sat, Mar 24, 2018 at 5:39 AM Greg Pakes [e-mail] wrote:

The reality is that the comment "it's open source, create your own fork",
is just not realistic. The overhead of maintaining your own fork and
keeping it up to date with the latest changes from the main repository
makes it rarely worth the investment. A much better approach is to petition
the maintainers and provide proper reasons as to why the feature is
important.

Sadly that will always result in issues like this one with some
dissatisfaction. I think the main issue here is that there doesn't seem to
have been a proper case put forward as to why this is a bad idea. The
argument "We should respect @shin- http:///shin-‘s response on this,
and believe there are good reasons for not implementing it." just isn't
going to satisfy people. The community need to see the "good reasons".

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/3574#issuecomment-375805478,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEV8Q9RseMxYS8KGyD9E2BK0pZv7OJpuks5thWt7gaJpZM4IyPQh
.

[gregpakes]

The point remains that pounding your fists and begging will rarely result in getting what you want

This is just not true. Whilst I don't condone it as a strategy to ask for free work to be done, it is actually quite effective. Its exactly how petitions work and the more people who "pound their fists" the more likelihood of people who matter taking notice. Again, I'm not saying I approve of it

There have been plenty of good use cases brought up in the thread as well

I can see one case, which essentially boils down to "you can do these two commands instead". But clearly people don't want it. So engage them and educate them as to why it is a good work around.

there’s no obligation from maintainers to do anything about it, nor give any reasons why

I agree. But you have to expect people to get frustrated with that. It's a dismissive attitude. At the end of the day, everyone here is on the same team. The people who maintain the project and the users of the project all just want the project to succeed.

[artisticcheese]

Which product is free exactly here? If I use docker-compose.exe and running docker EE, does not it really means that I'm in fact paying for product?

[ET-CS]

IMHO --build should pull; no need for any other flag or configuration. if you don't want it to pull then specify image version.

[lifeofguenter]

@ET-CS: image-versions are only tags and can still change to a different hash

[ET-CS]

good point @lifeofguenter thank you. can compose check if image changed to a different hash and pull in that case too?

In such scenerio, I think I'd want all the environments (dev, prod) to have the same image as it possible now that devs are using the new hash while production use the older.

I'd expect --build to bring everything to latest.

[agnjunio]

So, here's a good use case:

I implemented a CI for my microservices project, which pulls new images to a registry when we develop new features in the backend service. The frontend team (which knows little about docker) needs to have a way to bring up the entire backend stack on their local machines, and they rely on the most updated images. If something breaks, only then they will remember to pull the images.

Now this is what happened: An entire development sprint went to fail because someone forgot to update the backend images, and developed an entire feature based on an old version. Blame on frontend team, but this could be avoided with this functionality (which I'll be doing using wrapper scripts).

[shin-]

@agnjunio That sounds really unfortunate, sorry. However, if the person forgot to run docker-compose pull, I'm not sure how them forgetting to use a hypothetical --pull flag is any less likely?

[agnjunio]

@shin- Sorry, I forgot to mention an important thing: The solution in my case is to have the tag pull: always inside the yaml, maybe inside image: options

[solsson]

@ET-CS from https://github.com/docker/compose/issues/3574#issuecomment-382451356:

I'd expect --build to bring everything to latest.

IMHO --build should pull; no need for any other flag or configuration. if you don't want it to pull then specify image version.

AFAIK that would block the use case with a build using a FROM that is the result of a depends_on build.

version: "3.4"
services:
  some-base-image:
    image: our/base-image
    build:
      context: ./base
  # This Dockerfile has FROM our/base-image
  coolthing:
    depends_on:
    - some-base-image
    build:
      context: ./coolthing

I'm in favor of a flag as suggested in https://github.com/docker/compose/issues/3574#issuecomment-252861859 and https://github.com/docker/compose/issues/3574#issuecomment-279460839

[ET-CS]

@solsson
I am not sure I see why or what it's blocking in this use case.
please share more information about that if you can.

[ruanltbg]

@shin- The difference here would be that if I run docker-compose up --help I would receive a descriptive way in how to use the :latest image, instead nowadays I have to search in the doc or in the google that leads me to this thread and I need have to read all the comments to understand that docker-compose up does not do what I need/want, so now I need to run two command.

@agnjunio That sounds really unfortunate, sorry. However, if the person forgot to run docker-compose pull, I'm not sure how them forgetting to use a hypothetical --pull flag is any less likely?

Cordially

[ahannigan]

One of the main reasons we use docker-compose is the ability to place a docker-compose.yaml file in to a repository and have reproducible output when a developer pulls the repository and runs docker-compose up [service].

We use several services within our docker-compose files that perform tasks like running codegen and running a tool to dereference a JSON schema in to a single file.
Making sure these tools are up-to-date is critical especially if we update our codegen image to fix some common critical issue seen across all projects.

Having the ability to place:

services:
  codegen:
    image: myimage:latest
    pull: Always

would retain our ability to have a developer reliably run docker-compose up and get expected results, rather than having to supplement every single repository with documentation to remind developers to run a chain of commands or scripts to pull the latest availabe images before starting the apps.

It's not about "the functionality already exists to do this by running these commands", it's a better user experience.

Imagine when someone suggested adding --stop to docker-compose rm that the answer was "what's wrong with docker-compose stop myapp && docker-compose rm myapp, or if someone had requested the implementation of docker-compose down they were just asked why is docker-compose stop myapp && docker-compose rm -v myapp && docker-compose images -q myapp | xargs docker rmi ... impractical?

[jbauson]

This thread has been raised 2years ago and I still thought that adding a flag in the docker-compose.yml is the best way. In my case, we have like 37 services in a swarm, and updating 4-5 images from that total is not easy. I just created a shell for now that can monitor the change from the repository and do the pull for the specific image before recreating the service if it has been updated.

[jrynyt]

Another point here is that docker-compose up has a --quiet-pull option. When I was first trying to figure out if up included a pull, I assumed it did based on its presence. It goes to reason that not using --quiet-pull results in a verbose pull.

[eaglet3d]

Two years of people trying to convince the Docker Compose maintainers that having a --pull option would be a better experience without having to run a separate command. If the docker-compose maintainers just implemented the feature, to begin with, everyone's lives would be better. It seems to be clear that the current maintainers don't want to add this feature for the betterment of the community.

Maybe we should just fork docker-compose and update it ourselves.

[bean5]

If someone were to submit a PR, would it stands a chance if being accepted?
This is open source. I've been close to looking into implementing it a few
times myself. I assume it could be accepted, otherwise this role be closed,
right?

[ktravelet]

I ran into this same issue and holy crap, a bunch of whiners in here. This is FREE open source software. Give the maintainers a break. I'm sure they have much more important things to work on than this. If anyone needs this so badly, why don't you open a PR? If the code is clean with minimal risk, I don't see why they wouldn't accept it.

[dvdgsng]

This issue should be re-opened since the discussion lacks a good reason to not implement this request. People are more likely to start working on _open_ issues rather than closed ones.

[orodbhen]

The fact that this "closed" issue has remained so active does suggest that it has not been handled well.

Unfortunately, I've found that responses to issue posts on some GitHub repos are not very helpful. The tone is often terse, and suggests that feedback is less than welcome.

Some have pointed out here that this is an open-source project, and (at least most of us) aren't paying customers. However, it's also worth noting that submitting issue reports constitutes a significant time investment, even more so if you participate in the issue's resolution.

A user or developer, having spent time troubleshooting a problem, and found a workaround, will then decide if they want to spend additional time to report it. An unreceptive response from maintainers will likely result in them opting not to bother the next time around.

The software is not really "FREE", in the "free beer" sense. as we're all trying to participate in making it better. Having people willing to test your software and provide feedback is a valuable resource. Those with the requisite programming skills are even willing to contribute code, but they want some kind of indication that their contributions are welcome before spending time on it.

Obviously, comments that simply complain about a problem and demand that it be fixed aren't useful, but the majority aren't doing that, and comments like "it's open source, just fork it" are equally useless.

[jdalegonzalez]

@shin- Why was "--build" implemented? Is it different than docker-compose build && docker-compose up? Just trying to understand the philosophical different between --build (which got added) and --pull (which has been deemed redundant). Understanding the thought process might help me remember how things work. AND if the issue is opened, I'm happy to submit a PR. @everybodyelse... is it really the "spirit of opensource" that if you don't like something you fork? I thought the spirit of opensource was "if you don't like something you a) help contribute to the requirements if that's where you are, b) help contribute to the code if you can" and that you only forked if your requirements were very clearly something only you would benefit from. ie I thought we benefit most when we share - but I'm happy to be educated here.

[alonsohki]

After two years of insisting, giving good reasons that are ignored and enormous community support towards having this actually implemented, I would say that this feature won't make it just because @shin- doesn't want to. No reason to keep insisting.

[SuperSandro2000]

There is one reason: docker won't refresh the images if one pull fails and there is no good reason not to do it.

[c9s]

I am looking for the kubernetes image pull policy in docker compose, would be great if the "pull" can be used.

[Wenzil]

@shin- Stop being childish. Plenty of good reasons to implement this feature have been mentioned. At least be open to PRs...

[shin-]

You can disagree with me, but I'm disappointed that you would resort to ad hominems, @Wenzil. Our community generally holds itself to a higher standard.

[lig]

@shin- Our community mostly thinks the same keeping it unsaid because of the reason you've mentioned. @Wenzil is just being honest enough to say it out loud.
A lot of people I know prefer to not bother and gave it up on trying to convince docker-compose to move toward its users.

[ntwrkguru]

Many people disagree @shin- for very valid reasons described. At the very least, this should be a service declaration. Stringing bash commands together isn't a particularly good solution for programmatic deployments.

A lot of people I know prefer to not bother and gave it up on trying to convince docker-compose to move toward its users.

This. And it's not just docker-compose. Docker-stack, docker-machine, and docker-cli are all similar.

[lig]

https://github.com/docker/compose/issues/6316#issuecomment-438564025

[shin-]

Locking as this keeps getting derailed. We'll re-evaluate depending on the fate of https://github.com/docker/cli/pull/1498

[shin-]

As an update, we've decided internally to look into adding a pull_policy parameter to service definitions. We still have to figure out what the options and exact syntax will be, but we hope it'll satisfy the needs expressed in this thread.