Moby: Add support for specifying .dockerignore file with -i/--ignore

[tristanz]

As several people have mentioned (@thaJeztah, @duglin) in #9707, it would be great to be able specify the .dockerignore file using -i/--ignore in conjunction with named dockerfiles. It is often difficult to use named dockerfiles because the build context becomes too large.

[runcom]

dibs or sorry @tristanz are you already working on it?

[duglin]

To be clear, I didn't say I wanted it - just that it could be supported.
You may want to see if there's buy-in from the team first before you implement it.

[duglin]

@runcom ^^^^

[runcom]

roger that!

[tristanz]

I'm not working on about it.

It seems like a natural option given -f, since -f makes it much harder to use .dockerignore effectively. Without -i it is impossible to use -f for our repo because our build context is 1GB+. We have pretty classic setup: single repo with many services that share code. We'd like to be able to build all these services without custom scripting or slow builds.

[thaJeztah]

Not a maintainer (for this), I'd be +1 for this so that different Dockerfiles can exclude different parts of the build-context.

I'm not sure having a single-char flag (-i) is needed, perhaps --ignore-file (to match, e.g. --env-file).

@tristanz Perhaps you can describe some use-cases for this feature, it may help the maintainers make a well-founded decision whether or not this is something that's wanted

edit:
Our comments just "crossed", I see you just added some description on your use-case

[tristanz]

Yes, I believe our use case is exactly the use case described in the named dockerfile thread: a single repo with many services that share libraries. So this is just trying to make that same use case work for real setups. Single repos like this tend to be large (which we love for many reasons!), so without --ignore-file we can't use -f and are forced to resort to prebuild scripts.

[duglin]

@tristanz can you give a little more detail on these large repos? In particular, when I think about large repos with multiple sub-projects, I think to think of it as each project having their own sub-dir with their own Dockerfile but they all share a common dir that might be a sibling to each project. For example:

/common
/project1
/project2

and in these cases I wonder if supporting symbolic links wouldn't be easier. Meaning, /project1/common is a symlink to /common.

Also, to be clear, I'm not for or against a --ignore flag - still trying to understand the usecases.

[tristanz]

@duglin that's my setup exactly. I actually would _much_ prefer symlink support to solve this. It feels less docker specific, allows be to leave my dockerfiles unchanged (sub folder build context), and I can avoid writing lots of dockerignore files. I was just under the impression that named dockerfiles were the recommended alternative. I see symlinks as the better alternative by far.

[thaJeztah]

Are symlinks supported on Windows nowadays? (Wondering)

For my personal use-case, some containers are created using a volume to mount the source-code during development. Being able to exclude those files (huge _number_ of files, not the size per-se) when starting the dev container will save lots of time. (And, yup, there are workarounds)

[tristanz]

I guess the main question is whether the really is no solution for multiple container git repos with shared code. This seems like a extremely common use case, but any discussion of symlinks or relative paths is not finding traction as far as I can tell.

Here's the folder structure I'm thinking about:

/docker-compose.yml
/container1
   - Dockerfile
/container2
   - Dockerfile
/shared

Is there really no docker way? The best I have found so far is to simply copy shared into multiple places.

[tristanz]

Is it worth working on a pull request for this feature?

[thaJeztah]

Is it worth working on a pull request for this feature?

It hasn't been decided on, yet, but you could try. With the chance it gets turned down :)

Sometimes discussing a PR helps making a decision, so..up to you, I guess :)

[Ralle]

Another solution would be to look for the .dockerignore file where the Dockerfile is stored as the build directory is not necessarily the directory of which the Dockerfile is, but I would also prefer -i.

[thaJeztah]

Another solution would be to look for the .dockerignore file where the Dockerfile is stored

Unfortunately, that won't work if multiple Dockerfiles are in the same directory (e.g. Dockerfile.dev and Dockerfile.prod)

[Ralle]

Right. Then how about a Dockerfile command to describe the corresponding .dockerignore?

[thaJeztah]

I think the proposed solution (a --ignore-file option) is the best approach, that's most flexible. Only thing needed now is to decide if this is something that is desirable enough to implement.

[tristanz]

There are two thing I dislike about named dockerfiles even with dockerignore.

1. Dockerignore becomes needed everywhere.
2. Lots of containers don't have shared dependencies, but it's not clear to users of my repository what the build context should be. It would be fine if the root context was used everywhere, but our repository has forks of public Dockefiles that assume local context only.

Dereferencing symlinks seems much more elegant. I can always see all dependencies in each folder, like single Dockerfile repositories. Security could be handled not allowing symlinks to escape the outside the local folder unless an environmental variable or flag is set.

[tristanz]

I take back my second point. A better convention would be to put Dockerfiles at the depth of the build context. So Dockerfile.web rather than web/Dockerfile. This would make the context clear enough.

In terms of this whole issue, is there a reason why the build context is the entire folder not simply files and folders that are explicitly ADDed (plus Dockerfile/.dockerignore)? This would make multiple .dockerignore mostly unnecessary.

Dereferencing symlinks also looks like a non-starter because there is already a defined behavior -- create the symlink rather than dereference it.

[Ralle]

I make a web project and have billions of files. Having ADD for each would create an immense amount of intermediate containers and I would need a script that built my Dockerfile for me.

[tristanz]

@ralle I don't think anything would change. You just add a folder like normal. I'm just suggesting avoiding tarring and uploading things you never add.

[pikeas]

+1 to, in order of preference:

1) Build context shouldn't send all files, as described by @tristanz above. To maintain backwards compatibility, the CLI flag could be --full-context defaulting to true.
2) Symlinks followed during build.
3) Custom .dockerignore with -i per this issue.

[tristanz]

I think 1) in @pikeas list is by far the most elegant, but it is currently not easy to implement since parent images may have ONBUILD instructions.

[cusspvz]

@tristanz main difference between web/Dockerfile and Dockerfile.web is that on the first one you don't have the ability to add files behind that path.

Dockerfile is awesome for straight builds, but it lacks support for important deploy options such as environment-related diffs...

@fernandoneto please check if this works for you:

DOCKERPATH=/path/to/docker/build/folder
DOCKERFILE=Dockerfile.production
DOCKERIGNORE=.dockerignore.production
tar -cvzf - --exclude-from="$DOCKERPATH/$DOCKERIGNORE" $DOCKERPATH | docker build -f $DOCKERFILE -

[GordonTheTurtle]

_USER POLL_

_The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right._

The people listed below have appreciated your meaningfull discussion with a random +1:

@alenca
@cusspvz
@fernandoneto

[jakirkham]

Though I am not against having the .dockerignore file specified by an option, I find myself needing the following things:

• Being able to specify multiple .dockerignores (e.g. global, context, Dockerfile adjacent).
• Default behavior that looks in those places anyways.

[mishak87]

I would like to claim this one along with improvements to ignore internals.

[thaJeztah]

thanks @mishak87

I'll ping @tiborvass here to check if this will conflict with upcoming changes in the builder, but feel free to ask him on IRC #docker-dev (he's "tibor" on IRC)

[thaJeztah]

@mishak87 I just chatted with @tiborvass and it should not be a problem w.r.t. upcoming builder changes. Be aware that there's no "final" decision made yet for this feature, so we can discuss and decide during "design review" of your PR.

One thing; I'd opt for not using a short flag (-i) and only the long one (--ignore or --ignore-file). We try to preserve short flags for frequently used options (and we can always add a short version later).

Thanks in advance for your contribution!!

[mishak87]

@thaJeztah Thanks for the info, I will work on it the next week.

[thaJeztah]

Awesome! Thanks in advance @mishak87!

[mrluc]

This makes a lot of sense. Google led me here because I assumed that if you can specify a named Dockerfile you can specify a .dockerignore file.

For instance, if you do your building with a fat image from a Dockerfile.build, and then deploy against a skinny image from a Dockerfile, it makes sense that you'd like to have a .dockerignore.build as well that might ignore everything except the binary artifacts from the build step.

Only a minor annoyance given the size of my repos atm, but thank you @mishak87

LATER NOTE: because this comment has been here a while, I feel like I should update this to mention that even though this issue is legit, it's an _aesthetic_, 'dev UX' kind of issue.

If someone's stumbling across my comment in a StackOverflow kind of mood, because this is actually getting in the way of having separate build and run dockerfile/dockerignore pairs, don't forget that you can just put them in different directories. Ie, in your build script, after building against your 'build' Dockerfile/ignore and extracting the build artifacts, move the artifacts and the 'run' dockerfile/ignore into a separate toplevel (probably gitignored) binary_artifacts directory.

[pshomov]

+1

[jorge07]

+1

[5c077yP]

+1

[shanemcd]

Any updates on this?

[mishak87]

@thaJeztah I feel like my PR got stuck. Should I rewrite it to make it work with latest changes?
I could make time for it by end of September.

[jonywtf]

+1

[tkeeler33]

+1

[gittycat]

Let's assume 1) above where the whole context is not sent by default, couldn't the context then be derived from what's in ADD and COPY statements? No need for .dockerignore.

Alternatively (keeping things compatible, ie: sending the whole context by default), since a dockerignore seems to be specific to a Dockerfile, we could specify the dockerignore to use in the Dockerfile itself.

BTW, I am hitting the problem of the one .dockerignore with two Dockerfiles (one for building my app, the other for creating the production image). I will probably have to create a subdir just to build from the second Dockerfile. That's the simplest solution I can think of without a way to specify a different dockerignore for each Dockerfile.

[tarikjn]

@gittycat I like your approach, maybe both can be combined? ADD/COPY specify files to add to the context, .dockerignore what to ignore in them, seems like they have slightly different roles. This has the added benefit of improving performance and security of existing docker builds.

[johnt337]

Any traction on this?

[askoretskiy]

Having a single .dockerignore is annoying when you build multiple images via docker but really a problem once you use docker-compose. Then you cannot juggle with symlinks any more.

Moreover, docker-compose has a exactly this bug and cannot implement it till docker support this feature -- https://github.com/docker/compose/issues/2098

I would agree with other colleagues that .dockerignore would be really optional if docker generates context out of COPY and ADD commands from Dockerfile, ignoring all the files that are not there.

[whitecolor]

also --ignore option should take multiple ignorefiles

[bryanlarsen]

The lack of this feature prevents us from running docker builds in parallel while doing CI.

Our test scripts starts out something like this:

cp foo/dockerignore .dockerignore
docker build -f foo/Dockerfile .
cp bar/dockerignore .dockerignore
docker build -f bar/Dockerfile .
...

The builds could happen in parallel for a massive speedup. Slow CI is much less useful. :(

A common .dockerignore would result in a ~10GB context.

[thaJeztah]

This feature was put on hold, pending other changes that are being researched/worked on; one of those is making the builder "smarter" when sending the build context; see https://github.com/docker/docker/issues/31829

[enkoder]

Any update on when this will be worked on? This would solve a LOT of my problems when working in a monolithic repo.

[tzickel]

I have encountered this issue just today as well. I've spent time reading all of this discussion and most of the related PR.

My opinion is that there lots of radical ways to improve building in general, but today we have a very nice and simple way to declare some files to be ignored. There is just one glaring issue, while we can name the Dockerfile however we want (and thus allow building different dockers from the same directory), we cannot name the .dockerignore however we want and thus most of us end up sending tons of stuff to the context, that we don't need thus loading the network + build times unnecessary.

I do not understand why a simple PR to just add one option (which should have been done with -f), --ignore-file (even if it's one) conflict with future work on adding other ways to change the context (I assume that due to backwards compatibility .dockerignore will always be an option and thus people will always want the option to rename it).

[mxl]

@thaJeztah Could you reopen #18754? I see that you wrote that this feature was put on hold because of #31829 and other changes. But if you do not intend to remove ignoring files from build context with .dockerignore then I do not see how this feature conflicts with other changes. It's so frustrating that we are limited to .dockerignore file name and need to mess with scripts and using docker-compose is just wasting time and resources without option to use different .dockerignore files for several services with large shared build context. #18754 is only 32 lines of code and it could already help many people during these two years.
I agree with @tzickel - it's very small and very effective change that require no hard work and large proposals.

[cusspvz]

@thaJeztah could you please point which file handles the creation of the tar by the cli?

[TeraWilliam]

Sorry to spoil the party but I'm experiencing a similar problem and I suspect it's needed because of a bad repository design and layout. You'll need to track every new file addition and change multiple dockerignores, thus also asking you to do includes between docker ignores... and then templates... and god knows what.

Needing multiple dockerignores indicates a bigger issue at hand. Why is one not enough? Because architecture is improperly coupled.

[tonistiigi]

I propose a compromise that docker client would try to load <dockerfile-name>.dockerignore first and then fall back to .dockerignore if it can't be found. So docker build -f Dockerfile.foo . would first try to load Dockerfile.foo.dockerignore.

The advantage of this method is that it doesn't encourage creating Dockerfiles that only work when some specific options are passed to docker build. It also doesn't create any new flags that we would need to support. In the long run, this should be handled by the builder only pulling files it needs and granting access to files outside working dir(both become possible after #32677). Until that, we can provide a solution for people using different Dockerfiles in same source directory by reusing the -f flag that they are already setting.

[mxl]

@tonistiigi This sounds more complex and unintuitive then just using option to specify ignore file. You also need to respect path to Dockerfile because it does not have to lay in the same directory where corresponding .dockerignore file is located. Also there are cases when one can use the same Dockerfile with different docker-compose setups (and different .dockerignore files). Just adding option to specify .dockeringore is very simple to implement and gives maximum flexibility.

[tonistiigi]

@mxl unintuitive for who? That solution doesn't provide any knobs so there is no way to screw it up. If you think who should define the list of files needed/ignored for the build(-stage) it is the same person who writes the build definition(Dockerfile). Not the person who is invoking the command. They shouldn't even care that this info is in 2 files, it so only because of the limitation of the syntax of those files.

[whitecolor]

@tonistiigi I think both options should be included, and people would decide what works best for them, why not? I like your idea with convention over configuration.

[mxl]

@tonistiigi Ok, but still it limits to using one .dockerignore file per one Dockerfile. But as I said there are cases, especially with docker-compose, where using the same Dockerfile with the same build contexts but different .dockerignore files is required.

[tonistiigi]

@mxl Same dockerfile with different contexts? That seems like a quite weird case where it would be impossible for the reader to understand what is actually happening on build time. Can you show an example? For the case where multiple Dockerfiles use the same context, there are always symlinks and these cases should likely move to use build stages instead(compose support for that should be in next version).

[mxl]

@tonistiigi I do not have real example at hand, but imagine that you have directory with text files. Docker receives these files as build context except those that specified in ignore file passed as parameter to docker build. Started container runs bash script that randomly chooses word from of these files and prints it to output.
My main concern was that at the current moment we restricted to using single .dockerignore file per directory and if multiple docker-compose services use same directory as build context then they have a conflict because usually different services need different subset of files from that directory.
Also usually different services have different Dockerfiles (I did not yet meet real case when different .dockerignore files for the same Dockerfile are needed) so your solution will also work for me but I still think it's too restrictive and implicit. One should remember that .dockerignore file name should start with Dockerfile name and if he renames Dockerfile than he also have to rename .dockerignore file.

[iavael]

@tonistiigi building docker image with static files (css, js, gif etc) included (for development builds) and not (for release builds for stage/production with cdn).

[tarikjn]

@iavael this sounds like a weird setup, a build should generally be generated into an independent build folder down inside your project root for example, this is what you specify include with the COPY command in the Dockerfile for your release build. Including the whole tree and then adding ignores one by one is asking for trouble.

@tonistiigi what you say makes sense, I am agnostic to either approach but you have a point with

The advantage of this method is that it doesn't encourage creating Dockerfiles that only work when some specific options are passed to docker build

What is the status on pruning build context prior to sending it to the daemon based on a pre-parsing of the Dockerfile? Is there a separate discussion for this?

[tonistiigi]

One should remember that .dockerignore file name should start with Dockerfile name and if he renames Dockerfile than he also have to rename .dockerignore file.

There are no changes to the existing naming scheme. 99% of people should still use .dockerignore with any Dockerfile name. You only start to use new naming scheme if you want to make an exception for one of the files.

What is the status on pruning build context prior to sending it to the daemon based on a pre-parsing of the Dockerfile? Is there a separate discussion for this?

This would be a very small update once #32677 is merged.

[iavael]

@tarikjn for python projects it's something typical. You install system deps, copy in source code dir, run pip install -r requirements.txt and ready to go. I see that you have background in ruby, here should be something similar with bundler instead of "pip install".

[waterkip]

I would love to specify multiple dockerignore files in a build. My use case is that I have several .gitignore files:

1) $HOME/.gitexclude
2) $REPO/.gitignore (and you may have subdirs that also include a .gitignore)
3) $REPO/.git/info/exclude

While I would love for docker to include a function that respects the several gitignores, it can make my life easier if I could do this:

#!/usr/bin/env bash

git check-ignore -v **/* * | \
    awk '{print $1}' | \
    sort -u | \
    awk -F\: '{print $NF}' | \
    sed -e 's/\/$/\/**/g' | \
    >> .dockerignore.local

And then include the repo-wide dockerignore file and my own custom dockerignore file based on whatever gitignore files I have. Because I don't want to commit my local dockerignores into the main repo, that is why I make use of .git/info/excude and a general gitignore file for all my projects.

Perhaps support a syntax like git ls-files, it would make my day :)

[kyleaclark]

I only read through a portion of the replies, but chiming in to add this feature is also relevant to me for the same use case as described in the following SO post: https://stackoverflow.com/questions/40904409/how-to-specify-different-dockerignore-files-for-different-builds-in-the-same-pr.

I need a build for a deployed container (ignoring tests and other irrelevant files for prod) and also a separate container for running tests. Each instance will share the same Dockerfile but use unique docker-compose files specifying a binary value for whether to use a .dockerignore or preferably which dockerignore file name to apply.

[chrisLeeTW]

Just an idea.

Maybe we can change search logic of .dockerignore, that find file from same directory level of build context to same directory level of Dockerfile.

This change will make no different to original dockerfile codebase directory structures.

But also solve multiple sub-projects cases, with just put your .dockerignore in your sub dirs.

[leandro-lucarella-sociomantic]

@tristanz Perhaps you can describe some use-cases for this feature, it may help the maintainers make a well-founded decision whether or not this is something that's wanted

My use case is we have several projects and we are using docker just to build the projects for now. For that we have a .dockerfile. The build produces big files in a build/ directory, which is obviously ignored because to build the project we don't need the built files we are going to build.

Now we want to build images with the resulting binaries so we can use docker-compose to do a system integration test of all the projects. The runtime requirements are very different from the build requirements so we use a different Dockerfile for that,but for that we need the build/ directory to build the run image, that will ship the produced binaries.

Without having an option to select the .dockerignore file we need to either include the bulky build/ directory for the build image or include the bulky sources to build the runtime image. For now we'll probably write a wrapper script that just copy the Docker-build.ignore to .dockerignore and likewise with a Docker-runtime.ignore before building each image, but this solution is far from ideal.

It seems to make a lot of sense that if there is a way to pick a different dockerfile, there should be a way to pick a different dockerignore file too.

[aldencolerain]

@thaJeztah I've been reading through all of these issues regarding .dockerignore, build context, build volumes, build secrets, etc. It seems like this is one discussion spread over many different issues. Where is the best place to contribute to this discussion. I saw your comment https://github.com/moby/moby/issues/2745 about build kit. Is that where these discussions are moving forward?

I feel like being able to control the build context and providing the builder with things like ssh forwarding, secrets, etc would be such a huge step forward. Right now I many people just have to write scripts and hacks to work around conventions that make sense for public docker use but not for enterprise docker use. Are these issues ideological or technical?

[thaJeztah]

Not a _single_ answer to that; let me try to elaborate a bit:

BuildKit is a project where a completely new "build engine" is developed (https://github.com/moby/buildkit). The buildkit engine is providing low-level features to back a builder, which could be used to build from a Dockerfile (if integrated into Docker), and produce a Docker image, but could also be used to build source files, and produce a binary.

How those features are exposed when integrated into Docker is a separate discussion; there are optimisations that you'll get out of the box (e.g., smarter caching), but other features may still need to be "wired up" to expose them

[lironmo]

Any updates?
i also need this option.

the ignore file from same directory level of build context to same directory level of Dockerfile. +1

[NullVoxPopuli]

any progress?

[damianobarbati]

• for this... or any way to say --dockerignore=.gitignore

[aesadde]

@damianobarbati I think that for your case ln -s .gitignore .dockerignore will do :)

[SamuelDeal]

I need this because of multiple build contexts (and in docker-compose also, so the 'ln -s' solution doesn't work for me)

[jingqi]

I need this so much.
Can't image why we have -f option but left --ignore-file out.

[aaparmeggiani]

1+ for --ignore-file

or, even better when the context is not needed at all, --no-context (??)

[jonaskello]

I have a monorepo which needs to build multiple images, but I can only have one .dockerignore file so it becomes difficult to make the build effective. Given the rising popularity of monorepos this issue seems even more useful.

[gregpakes]

@jonaskello

I have exactly the same issue. Mono repo with 6 dockerfiles. Context sent to each is > 8GB. :-(

[coretemp]

I would like to have this feature, because the .dockerignore feature doesn't meet its specification (i.e. is broken).

Additionally, I would like to point out that the .dockerignore feature as it currently exists is a complete design failure (as evidenced by everyone complaining about it). It's not consistent with the other CLI options such as -f.

I have a file .dockerignore with just .git as the contents in the context directory, but there is no difference in image size with or without the .dockerignore file (which should be impossible). I am using an absolute path to the context directory in case that's relevant which is different from the current working directory.

Please let me know alternatively when you have added a regression test.

[tzickel]

https://gist.github.com/tzickel/2111dd013d1c4f8c765a538e74646f95

If you don't mind using python and the .gitignore syntax (unless someone feels like adding .dockerignore support to python's pathspec library, although they are not very different) this script can handle multiple paths for .gitignore files to ignore from the base context path (also can show you which files are sent). Don't forget to read the comments at the top of the script.

[DavidUser]

When you had many docker builds over the same repo the .dockerignore should contain only what should be ignored on all builds, but it isn't a good practice considering that at future should have new dockerfiles that add the entire repo and need some file previously ignored.
e.g.: I had a build container and a run-time container, some things should be carried on build container and shouldn't on run-time container.
We need a way of cascade hierarchy of .dockerignore, add a ignore dockerignore entry seems really a crazy workaround.

[thaJeztah]

https://github.com/moby/moby/issues/37129 may be a good solution for situations where multiple Dockerfiles have to be built from the same source/repository

[mitermayer]

Whats the current workaround ?

[robertveloso]

Vote for --ignore

[tzickel]

@mitermayer https://github.com/moby/moby/issues/12886#issuecomment-430502559 is an option.

[falnos24865]

@mitermayer #12886 (comment) is an option.

That is a really hacky workaround. Adding a flag to allow a different name/path for the docker ignore file is really needed. We have different context needs for our production and local dev builds. Currently we have to waste time/space building the full production build locally, even though we are going to mount most of the stuff.

While I am not familiar at all with the Docker codebase I can't see this being a hard thing to add, and it certainly has its benefits. Is it not considered worth adding at all, or is it just a very low priority?

[robertveloso]

I'm using the gist for the job, but I do think it would be better if the flag was built in as falnos commented.

[mathieulj]

Another (one liner) work around I've used is manually bundling the context with tar and piping that to the build:

# Manually build our context... this is hacky but docker refuses to support symlinks
# or selectable .dockerignore files
tar ch --exclude-from=.dockerignore * | docker build Dockerfile

[mildsunrise]

The solution of this comment (PR above) was merged and has landed in Docker 19.03! :sparkles:

You need to enable BuildKit mode to use it, example:

~~~ bash
$ export DOCKER_BUILDKIT=1

$ echo "FROM ubuntu \n COPY . tmp/" > Dockerfile
$ cp Dockerfile Dockerfile2
$ touch foo bar
$ echo "foo" > .dockerignore
$ echo "bar" > Dockerfile2.dockerignore

$ docker build -t container1 -f Dockerfile .
$ docker build -t container2 -f Dockerfile2 .

$ docker run container1 ls tmp
Dockerfile
Dockerfile2
Dockerfile2.dockerignore
bar
$ docker run container2 ls tmp
Dockerfile
Dockerfile2
Dockerfile2.dockerignore
foo
~~~

[mildsunrise]

@tonistiigi Any estimates on when the feature will be ported to the default builder as well?
Would a PR to docker/cli be accepted?

[tonistiigi]

@jmendeth Feature development in legacy builder is effectively frozen according to roadmap. If the PR is simple and doesn't add maintenance cost it would probably still be accepted. It might be somewhat more tricky to implement this for legacy builder though as there are some edge cases where ignored files are put in context and removed on the daemon side.

[tonistiigi]

As https://github.com/moby/buildkit/pull/901 is frontend only it works in current versions of Docker as well with an updated #syntax. There are no release versions yet but # syntax=docker/dockerfile-upstream:master # syntax=docker/dockerfile-upstream:master-experimental should work.

[LinusU]

Would be very nice if we could also get a flag (e.g. --ignore) to specify whichever file we want... ☺️

[mildsunrise]

@jmendeth Feature development in legacy builder is effectively frozen according to roadmap. If the PR is simple and doesn't add maintenance cost it would probably still be accepted. It might be somewhat more tricky to implement this for legacy builder though as there are some edge cases where ignored files are put in context and removed on the daemon side.

I see, thanks!

[virgofx]

Will this be resolved in next stable release?

[ddnomad]

+1 for this one (gosh, 4 years old issue).

We have two docker files (Dockerfile.dev and Dockerfile.prod). Development tooling mounts a volume with a source code to an already built container which is handy for coding, but as we can't ignore node_modules in a production image we are forced to push them to a development image as well (during build) which adds ~200mb of overheads (and is completely unnecessary).

Would be reaaally nice to have that .dockerignore.dev and .dockerignore.prod as separate files.

[virgofx]

@ddnomad Same use case here. Development dockerfiles dont need to seed any context, production seeds app during CI build time.

Don't mind setting minimum Docker Desktop version for developers and in CI -- just wondering how to get it working and what required version. In development currently using:

[ddnomad]

@virgofx For our use case it is not the end of the world yet (till context will get over 1GB lol).

I think sucking it up and spending some time to dynamically generate .dockerignore during different types of a build might make sense in the long term tho. Can be a part of make target or something.

Terrible hax, but will suffice I recon. Obviously would require to keep .dockerignore git ignored, otherwise that might cause a mess in VCS.

Something like this (+prod version which copies .dockerignore.prod to .dockerignore).

.PHONY dev-docker-build
dev-docker-build:
    cp .dockerignore.dev .dockerignore
    docker build . \
       -t $(DOCKER_TAG)-dev-$(DOCKER_VERSION) \
       -f Dockerfile.dev

[tonistiigi]

see https://medium.com/@tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae for how to switch between different set of files with build args

[virgofx]

see https://medium.com/@tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae for how to switch between different set of files with build args

Issue is in regards to a request for a unique parameter to specify separate .dockerignore files ... not multi-stage build patterns or parameterized builds.

[virgofx]

@ddnomad Yep --- that would work ... not very clean. I thought they already added this into the baseline? Just wondering when it's available?

[ddnomad]

Yep --- that would work ... not very clean

@virgofx viva la hax 🤣

Just wondering when it's available?

Around the time when docker images ls will stop outputting empty table I recon 🌚

[tonistiigi]

@thajeztah I think we can close this as the use-cases have been addressed in 19.03 and there is a lot of outdated/incorrect advice in here that may confuse new readers.

This has been addressed by three features:

• https://github.com/moby/moby/issues/12886#issuecomment-480575928 allows setting a dockerignore file per Dockerfile if the repository contains many. (Note that this was the exact description for the initial issue)
• BuildKit automatically ignores files that are not used, automatically removing the problem where different sets of files needed to ignore each other.
• The cases where same Dockerfile uses different sets of files for different "modes" of build (eg. dev vs prod) can be achieved with multi-stage builds and defining the mode changes with build arguments.

This is not the same syntax originally requested in here because that version is incompatible with stability guarantees users expect from Dockerfiles. The provided solutions solve the problems for the users who need better control for ignored files, without adding new ones to the others.

[marcovergueira]

Would be great to have this feature, in my scenario I have 4 different images to build in a pipeline, however I have a monorepo and it's mandatory to use as base path the root of the repo.

Already tried to generate the .dockerignore dynamically before build each image, but it's a pain to sync the 4 parallel build stages because they use the same base path at the same time, and it's a mess when try to get the right .dockerignore file.

[mildsunrise]

@marcovergueira What's the problem? You can specify a dockerignore for each Dockerfile... The feature is already there

[thaJeztah]

I'll close this issue because I think all use cases should be addressed https://github.com/moby/moby/issues/12886#issuecomment-518843764

I _highly_ recommend enabling buildkit, as it addresses quite some performance issues related to uploading the build context (even without having to use a .dockerignore file);

enable buildkit by setting the DOCKER_BUILDKIT=1 environment variable

[dm17]

export DOCKER_BUILDKIT=1

This feels hacky... Why can't we just have 1 dockerignore file per dockerfile? There's already 1 dockerfile per directory by default. This does not behave rationally imo... Each service that is built with docker-compose --build should use the dockerignore file that is in the dockerfile's build context.

[dm17]

@thaJeztah Does this mean I can do the following:
sudo docker build --no-cache -t specific_img -f Dockerfile-specific_img .

And when building specific_img, it will use the file called "Dockerfile-specific_img.dockerignore" to determine which files to omit from specific_img's build?

I couldn't find this in the docs anywhere... @jmendeth's comment isn't exactly clear to me with all of the echoing stuff.
Thanks!

[mildsunrise]

Does this mean I can do the following:
sudo docker build --no-cache -t specific_img -f Dockerfile-specific_img .

And when building specific_img, it will use the file called "Dockerfile-specific_img.dockerignore" to determine which files to omit from specific_img's build?

Yes. If you do docker build <file> then <file>.dockerignore will be used instead of .dockerignore if it exists.

This feels hacky...

It will be the default someday, if it isn't already.

[dm17]

Does this mean I can do the following:
sudo docker build --no-cache -t specific_img -f Dockerfile-specific_img .
And when building specific_img, it will use the file called "Dockerfile-specific_img.dockerignore" to determine which files to omit from specific_img's build?

Yes. If you do docker build <file> then <file>.dockerignore will be used instead of .dockerignore if it exists.

This feels hacky...

It will be the default someday, if it isn't already.

Default meaning no "export DOCKER_BUILDKIT=1" required? When using "export DOCKER_BUILDKIT=1" - do you recommend putting that into our Dockerfile?

[thaJeztah]

No, no need to put it in the Dockerfile; you can set it in your shell (or, for example, in your .profile) to enable buildkit for your builds

[matteocontrini]

Is there a plan for when the buildkit implementation is going to be the default?

[tonistiigi]

@matteocontrini after https://github.com/moby/moby/pull/38738

edit: note that BuildKit can be currently set default on daemon level as well if you don't like setting the env.

[Graham42]

As of October 10th 2019, here's the behaviour I've observed (I couldn't find it documented anywhere)

• docker build -f Dockerfile2 will use .dockerignore if no Dockerfile2.dockerignore exists.
• docker build -f Dockerfile2 will use Dockerfile2.dockerignore if it exists. It does not combine the contents of the 2 files. ie Dockerfile2.dockerignore does not inherit what is in .dockerignore

[tzickel]

Just reminding if anyone is interested in this script:
https://gist.github.com/tzickel/2111dd013d1c4f8c765a538e74646f95

[jamiejackson]

Is this feature documented anywhere? I'm having trouble using it in docker compose.

---

Update, with regard to compose: From what I can tell, using this from compose requires putting a few different things together:

• Need to enable buildkit, which also requires the daemon.json to have experimental enabled. daemon.json:

...
   "experimental": true,
   "features": { "buildkit": true }
...

• Need to run docker compose in CLI mode
  
  
  
  • set environment variable COMPOSE_DOCKER_CLI_BUILD=1
  
  
  • available in docker-compose as of 1.25.0-rc3. Released October 28, 2019 - https://github.com/docker/compose/pull/6865#issuecomment-546929648
  
  

[thaJeztah]

@jamiejackson yes, unfortunately, the docker-compose feature is not yet in a release candidate; I asked some of the compose maintainers if there's plans to cut a 1.25.0-rc3

[nok]

@Graham42 , maybe here? https://docs.docker.com/engine/release-notes/#19030

Look for Dockerfile specific dockerignore file (for example, Dockerfile.dockerignore) for ignored paths. docker/engine#215

[guillaume86]

@thaJeztah What about windows users? Buidkit seems great but it's not available on windows and support seems very far away. That missing feature can be quite annoying as a lot of people discussed earlier, it basically makes docker-compose unusable in my case (big monorepo).

Should I make a new issue or do we discuss it here?

Would backporting that single feature from buildkit to the legacy build system (so windows) under some specific environment flag be acceptable?

[mildsunrise]

@guillaume86 To quote @tonistiigi:

Feature development in legacy builder is effectively frozen according to roadmap. If the PR is simple and doesn't add maintenance cost it would probably still be accepted. It might be somewhat more tricky to implement this for legacy builder though as there are some edge cases where ignored files are put in context and removed on the daemon side.

[jonathan-marcus]

@tonistiigi I have a pretty reasonable use case for -i/--ignore-file that is not covered by the new features in 19.03: specifying a Dockerfile from stdin. For security the code I am containerizing is mounted on a read-only filesystem. I pass my Dockerfile via stdin, but I have no way at all to provide a Dockerignore file. An command line argument for specifying a Dockerignore would be a simple and easy fix for this.

[tonistiigi]

@jonathan-marcus hmm, if you have no way for writing files for your dynamically created dockerfile and need to use stdin for that how would you write an ignore file what matches your Dockerfile what you could point to with the new flag? I think it is not unreasonable to expect temporary files can be written to the system. In fact -f - is not guaranteed to work on a system where temporary files can't be created. At least in some cases the way it works is to write stdin to /tmp and then share it from there to the build daemon. There's an issue where for extended security we'll probably start to do this temp copy step for all of the builds in the future, even without -f -.

[jonathan-marcus]

Most of the file system is writeable; the part that contains the build context is the only part that is mounted read-only. So I am capable of writing temp files anywhere. I would love to be able to pass in --ignore-file /tmp/dockerignore. I just can't write that dockerignore file anywhere in the build context.

My use case: I'm deploying a program that (among other things) containerizes user's code. It itself runs in a container and uses Docker-in-Docker to access the user's file system. I do a read-only mount of the target path on the external file system, and then my program runs docker build from within the container.

[tonistiigi]

@jonathan-marcus Then you can just

d=$(mktemp -d)
cat - > $d/Dockerfile
mv /tmp/dockerignore $d/Dockerfile.dockerignore
docker build -f $d/Dockerfile .

[jonathan-marcus]

Clever! That will work really nicely, thanks. I'll try it out and let you know how it works.

[guillaume86]

Feature development in legacy builder is effectively frozen according to roadmap. If the PR is simple and doesn't add maintenance cost it would probably still be accepted. It might be somewhat more tricky to implement this for legacy builder though as there are some edge cases where ignored files are put in context and removed on the daemon side.

@tonistiigi So I took a look into adding .dockerignore support to the legacy builder and I have some questions, could you help?

Changes are required in "docker/cli", but depending on a choice changes might not be required in moby/moby. Here's my two options:

• in the cli, send the pathToDockerfile.dockerignore with the build context and also handle it builder side, so changes also required here, and hiding the feature behind a ENV variable seems complicated (need to pass info from cli to docker)

• in the cli, rename pathToDockerfile.dockerignore to .dockerignore in the tar archive (if it does exists), no changes required here and hiding behind an env var is trivial

Which way do you think I should go?

[fabb]

It seems not to work when the Dockerfile contains the line # syntax = docker/dockerfile:1.0-experimental - is this a known limitation or a bug?
If this line is present, always the regular .dockerfile is used.

UPDATE: Interesting, when I use # syntax = docker/dockerfile:experimental (no 1.0), it works correctly.

[tonistiigi]

@fabb This was added after 1.0 https://github.com/moby/buildkit/pull/901/commits/7ac4bf072f86891203a05f46f9de16f68fc2ffe6 shows the tags that have it.

[rulatir]

OK, now that the feature is present, please document it exhaustively, with complete examples for different combinations of the following variables:

• location of dockerfile
• name of dockerfile
• build context root

and demonstrating correct answers to the following questions:

• where should the dockerignore file be placed?
• how should the dockerignore file be named?
• how (i.e. relative to what base path) should the paths in the dockerignore file be specified?

[ferrastas]

If you take a look at the MR from the last @tonistiigi message (https://github.com/moby/buildkit/pull/901) in the description and tests you will see how the file/s should be named and where to be placed.
Instead of demanding an exhaustive documentation with all your questions answered, you can cooperate with the project and do the exercise by yourself and share all your finds in this same thread.

As mentioned in the repo description this is a collaborative project, we can all help to make this project better :)

[numToStr]

It is not a bad thing for asking about the documentation. And no one is willing to dig through the issues and PRs to find the solution. And also It is relevant enough to be included in the documentation, so that everyone can have access to it with some clear examples.

[rulatir]

Instead of demanding an exhaustive documentation with all your questions answered, you can cooperate with the project and do the exercise by yourself and share all your finds in this same thread.

Nonsense. Reverse-engineering undocumented functionality from issues and PRs is chasing clouds. As long as the functionality - either its existence or exact semantics - isn't documented, there is no guarantee that the result of this effort will even still be valid next morning. Undocumented functionality, or undocumented aspects of functionality, can change any moment. They cannot be relied on or used in serious projects. Documentation is not merely "here's how this works". It is also "we promise it will continue working as described", and that promise can only be made unilaterally by the authors of the software. This aspect of the project is not collaborative.

[rulatir]

If you take a look at the MR from the last @tonistiigi message

What's a MR?

The Build method where dockerignore file paths are constructed and used is a 344-LOC monster in a programming language I have never coded in, using complicated APIs I am completely unfamiliar with. I shouldn't have to parse this code.

[kamazee]

What's a MR?

"Merge request," I guess. Synonym of "Pull Request" in GitLab and Bitbucket.

[jonathan-marcus]

@tonistiigi I finally was able to try your suggestion from April and it worked fine on my machine (Docker 19.03.12 on Mac). From my coworker's machine (Docker 19.03.12 on CentOS) it did not work.

Our setup:

/dir/.dockerignore
/dir/.git
/dir/<lots of other contents>
/tmp/Dockerfile
/tmp/Dockerfile.dockerignore

Command: DOCKER_BUILDKIT=1 docker build -f /tmp/Dockerfile /dir

The .dockerignore in the build context (/dir/.dockerignore) excludes the .git directory with a **/.git line. My custom .dockerignore (/tmp/Dockerfile.dockerignore) is a clone of the original, plus an override !**/.git line at the end.

My (desired) result is that the resulting image has the .git directory. The same command from my coworker's computer (undesirably) produces an image that does not have it.

Is this feature supposed to allow me to override the .dockerignore that is already in the build context? Do you have any ideas why I would get different results from my coworker? I used docker --version to check the version, but I suspect there are other relevant pieces that I didn't check.

I can work on a more minimal reproducible example and possibly file a new issue, if warranted.

[thaJeztah]

Is this feature supposed to allow me to override the .dockerignore that is already in the build context?

Yes, theDockerfile.dockerignore will be used instead of the default .dockerignore

I used docker --version to check the version

Make sure to check the output of docker version, because docker --version will only show the version of the docker CLI, not of the daemon

[jonathan-marcus]

Thanks, I had a feeling I was missing something obvious with the version.

I'm on:

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false

My coworker is on:

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:06:30 2019
  OS/Arch:          linux/amd64
  Experimental:     false

What's the minimum version needed for this feature? I'd like to autodetect this and alert my users to upgrade. I'm guessing 19.03, given that https://github.com/moby/buildkit/pull/901 was merged in March 2019.

Side note: my coworker is surprised that dnf on CentOS isn't giving him a notice to upgrade Docker, given that he's a major version behind.

[thaJeztah]

What's the minimum version needed for this feature?

Yes, that would be Docker 19.03.x. Older versions of docker reached EOL, so are no longer maintained.

Side note: my coworker is surprised that dnf on CentOS isn't giving him a notice to upgrade Docker, given that he's a major version behind.

Interesting; not sure why newer versions wouldn't show up

[jonathan-marcus]

I can confirm that upgrading fixed this. Thanks for the help!

[philipbjorge]

@rulatir -- Thank you for advocating on behalf of the users.
I'm still chasing clouds here, but I hope to catch them shortly :upside_down_face:

[simonefuoco]

The solution of this comment (PR above) was merged and has landed in Docker 19.03! ✨

You need to enable BuildKit mode to use it, example:

$ export DOCKER_BUILDKIT=1

$ echo "FROM ubuntu \n COPY . tmp/" > Dockerfile
$ cp Dockerfile Dockerfile2
$ touch foo bar
$ echo "foo" > .dockerignore
$ echo "bar" > Dockerfile2.dockerignore

$ docker build -t container1 -f Dockerfile .
$ docker build -t container2 -f Dockerfile2 .

$ docker run container1 ls tmp
Dockerfile
Dockerfile2
Dockerfile2.dockerignore
bar
$ docker run container2 ls tmp
Dockerfile
Dockerfile2
Dockerfile2.dockerignore
foo

Hi, this solution works but do not work with docker-compose commands.
When you use a custom Dockerfile (ex. Dockerfile-PROD) and a custom .dockerignore (Dockerfile-PROD.dockerignore) and link the Dockerfile name in docker-compose.yml, Dockerfile-PROD.dockerignore cannot be read.