Moby: Force clean build of an image (without cache)

[matthewmueller]

Sometimes commands return exit 0 but end up in a failed state. Right now there's not a good way of reseting the cache, so it tries the command again.

[crosbymichael]

Would something like docker build --no-cache . work for you?

[matthewmueller]

yep that would be great.

[soichih]

Is there a way to remove only the specific cache? Like.. I have 100 steps in my build script, and I want to only redo the step 80.. but I don't want to redo other steps.

[thaJeztah]

I don't think it will be possible to _only_ rebuild step 80, because steps 81..100 will build upon that.

However, only slightly modifying a step should invalidate the cache, for example;

RUN echo "cache-bust" && your original steps here

I think that should re-build steps 81..100. Haven't tested it, but worth a try

[soichih]

Ah, thank you for the idea.

This worked.

RUN yum -y install firefox #redo

So it looks like Docker will re-run the step (and all the steps below it) if the string I am passing to RUN command changes in anyway - even it's just a comment.

[thaJeztah]

Great that you got it working!

Just wondering; how did you get on _100_ steps/layers? That's quite a lot, it might be an idea to group them together in logical groups; Also, you can specify multiple instructions per RUN line, just as you would in the shell, e.g.

# Install Foo and related (wrapped to multiple lines, just for readability)  
RUN yum -y install package1 \
                   package2 \
                   package3

# Install Bar and related    
RUN yum -y install package4 package5 package6

# Configure Bar and related (multiple commands, separated with ;)
RUN sed -i -e 's/foo/bar/'  /some/config.cnf ;\ 
    sed -i -e 's/re/place/' /some/config.cnf ;\
    echo "hello world" > /some/other/config.cnf

[soichih]

Oh, I actually don't have any Dockerfile that has 100 steps in it (at least not yet..) I used "80 out of 100 steps" to illustrate that if I do have such Dockerfile, it will be important to be able to rebuild only the portion of the build.

Thanks for your suggestions!

[thaJeztah]

Ok then! I took the 80/100 steps literally, because, well, you never know what somebody is trying to achieve :smile:

[RichardBronosky]

@soichih, @thaJeztah, Thank you so much I found this by Googling "docker build expire cache" and I only wanted to expire a specific hash. I got the explanation of why I actually needed to expire subsequent hashes and a perfect example of how to do it. I didn't want to do docker build --no-cache just because I changed one line in my github project. This allowed me to add a mere 3 seconds to my 4 second build time rather than the full 8 minutes required to build with no cache. This gets me a feedback loop I can live with! Automate all the things!

[manuelpayet]

Here are my two cents: the docker cache is used only, and only if none of his ancestor has changed (this behaviour makes sense, as the next command will add change to the previous layer).

The cache is used if there isn't any character which has changed (so even a space is enough to invalidate a cache).
So, it's not possible to clear the 80th step cache, and keep using 81th step cache to 100th step cache.

A change of a cache will invalidate all the subsequent cache layers.

[snackycracky]

I would have thought one can use something like this: docker build --commit 3e1670c59af1 ... to invalidate the cache from this point on and rebuild everything after this.

[thaJeztah]

@snackycracky docker 1.9 adds "build time parameters", which can be used to achieve that; see http://docs.docker.com/engine/reference/builder/#arg

Doing something like docker build --build-arg commit=3e1670c59af1 .. would set the commit arg, which can be used to (e.g.) use a specific commit

[snackycracky]

:+1:

[vhmth]

Thank you for building --no-cache @crosbymichael.

[Soufraz]

@crosbymichael --no-cache is perfect! Thank you.

[jcoffland]

@manuelpayet I think everyone knows that invalidating one step will invalidate all subsequent steps. That's precisely the feature I'd like to have. For example, it would be very helpful if you could do something like this:

docker invalidate f0d6ceea004e

[wedesoft]

:+1:

[Niondir]

is there a solution?

[blackwellops]

@Niondir yes --no-cache

[zinking]

@jcoffland I saw this is closed, is your suggestion taken ? or did you create a separate request to track this ? this is especially helpful for places where network download is slow (like my region).

is it very difficult to implement? I'm not familiar but sounds like just delete a portion in the local cache.

[jcoffland]

@zinking no it is not possible to invalidate one step in the build as far as I know with out digging in /var/lib/docker.

[ajrowr]

Since Docker checks the files that are being COPY'ed for changes, this is what I'm using -
In Dockerfile, straight before the stage I want to cachebust:
COPY .cachebust /tmp/cachebust
In bash:
date > mycontainer/.cachebust && docker build mycontainer
This way I only need to change the Dockerfile to move the COPY if I want to uncache a different stage.