Moby: Proposal: Dockerfile add INCLUDE

[dysinger]

[shykes]

+1

[keeb]

+1

[ptone]

Yes this would be great to see +1

[jpfuentes2]

I think this would be a great feature as I want to leverage some of my knowledge/experience with systems like Chef whereby you can compose complex builds using smaller/simpler building blocks.

Does anyone have a way they're implementing this now?

[crosbymichael]

Can someone give me a few examples on how they would use this? ping @ptone @keeb @dysinger

[ptone]

Lets say I have build file snippets for different components of web architecture.

I may want to include 1 or more of them on a single container image - or in general bundle things that always go together.

Simple examples might be nginx and varnish always go on the same container, or redis and pyredis.

I might have a "Scientific Python" list of python packages and linked libs, that I may want to include in other images.

The problem with FROM and base images is that you can't remix things the same way you can with includes. A base image is all or nothing - if you don't want something, your only hope is that you can go back to a 'lower' base image, and re-add stuff manually that you want, skipping the stuff you don't.

It is essentially a case of inheritance vs composition in many ways.

[binocarlos]

+1 to this - @crosbymichael @ptone @keeb @dysinger @shykes

I am at this exact point where I have an appserver base image (which is just node.js).

I also have a collection of Dockerfiles representing little bits of services that have deps - so:

ImageMagick:

from appserver
run apt-get install imagemagick

RedisSession:

from appserver
run apt-get install redis-server

To have a container that is both ImageMagick and RedisSession:

from appserver
run apt-get install imagemagick
run apt-get install redis-server

Whereas the following syntax means I can build up a folder of modules and include them by name in the application Dockerfile:

from appserver
include ./modules/imagemagick
include ./modules/redis-server

Now, because Docker is so darn brilliantly good : ) this is currently trivial to do (i.e. read module file and inject here) and so I'm not sure if for me this is a required feature.

However - it would make a user app's Dockerfile (i.e. the last thing in the chain for this scenario) much more about composing modules together than about creating a strict inheritance tree where some combinations are not possible - my 2 cents on what is a joy to work with otherwise :)

[Thermionix]

+1 would also be good to reference out generic blocks (possibly to a github raw link?)

[frankscholten]

+1

[joelreymont]

I'll re-implement this on top of #2266.

[prologic]

+1 Turns Docker and the Dockerfile into a portable rudimentary configuration management system for the constructions of portable Docker images :)

[ghost]

This would help a lot. +1

[deeky666]

+1

[peenuty]

+1

[newhoggy]

+1

[jfinkhaeuser]

Not to sound negative here (pun intended), but -1.

I completely get the use cases for an include statement. But then I also get the need for parametrized Dockerfiles, and then for conditionals, etc. Continue down that path, and you'll end up implementing a programming language in Dockerfiles, which may even become turing complete. The cynicism in that statement is free of charge, by the way ;)

Why not use a preprocessing step instead? You don't even have to program your own, you could use m4 (used in autotools for this purpose) or the C preprocessor (used in IMake, which does a similar job as autotools but is pretty much defunct these days).

Makefile:

Dockerfile: Dockerfile.in *.docker
  cpp -o Dockerfile Dockerfile.in

build: Dockerfile
  docker build -rm -t my/image .

Dockerfile:

FROM ubuntu:latest
MAINTAINER me

#include "imagemagick.docker"
#include "redis.docker"

Run make and it'll re-build the Dockerfile if any input file changed. Run make build and it'll re-build the Dockerfile if any input file changed, and continue to build the image.

Look ma, no code!

[prologic]

On Tue, Mar 11, 2014 at 6:45 PM, Jens Finkhaeuser
[email protected]:

Not to sound negative here (pun intended), but -1.

I completely get the use cases for an include statement. But then I also
get the need for parametrized Dockerfiles, and then for conditionals, etc.
Continue down that path, and you'll end up implementing a programming
language in Dockerfiles, which may even become turing complete. The
cynicism in that statement is free of charge, by the way ;)

Why not use a preprocessing step instead? You don't even have to program
your own, you could use m4 (used in autotools for this purpose) or the C
preprocessor (used in IMake, which does a similar job as autotools but is
pretty much defunct these days).

I'm in agreement with this. Having toyed with design and implemtantions of
a few languages myself over the years
turning Dockerfile(s) into a "scripting" language even if it's not turning
complete sounds like something that Docker should not do.

As Jens clearly points out there are better more appropriate tools for this
job.

cheers
James

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

[hunterloftis]

+1

The slippery-slope argument for a turing-complete scripting language in Dockerfiles seems a bit extreme. INCLUDE (or FROM ./relative/path) just lets you create a common base image locally so you can reference a file system (for example, in your app's repository) instead of relying on the Docker registry for what should be a self-contained app definition.

[prologic]

On Wed, Apr 9, 2014 at 12:15 PM, Hunter Loftis [email protected]:

The slippery-slope argument for a turing-complete scripting language in
Dockerfiles seems a bit extreme. INCLUDE (or FROM ./relative/path) just
lets you create a common base image locally so you can reference a file
system (for example, in your app's repository) instead of relying on the
Docker registry for what should be a self-contained app definition.

I don't agree with this. The very notion of referencing and building a base
image is already there
and it only accesses the public registry (or a private onf if you're so
inclined) if you don't have said image.

I'm still -1 on this -- it adds more complexity for little gain. I'd rather
see Docker pick up some YAML-style
configuration format for "configuring one or more containers" ala fig et
all.

cheers
James

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

[jfinkhaeuser]

The slippery-slope argument stems from experience with a bunch of other DSLs. There's a general trend for DSLs to become turing complete over time.

The include statement in itself presents little danger here, but consider that in almost every language, include or import statements are linked to the concept of an include path, quite often set via an environment variable.

There's a reason for that: having include statements means you can collect building blocks into reusable libraries, and having reusable libraries means you'll want to use the same building blocks in multiple Dockerfiles. The best way to do that is to be agnostic to the actual file location in your include statement, but instead have a "canonical" name for a building block, which is usually a file path relative to some externally provided include path.

So what's wrong with that? Nothing, except (and I'm quoting you here):

(...) instead of relying on the Docker registry for what should be a self-contained app definition.

I agree. A Dockerfile should be a self-contained app definition. Once you include stuff from any location that's shared between projects, though, you have anything but that - the needs of one project will lead to modifications of the shared file and those may not reflect the needs of the other project any longer. Worse, any kind of traceability - this version of the Dockerfile should build the same image again - is completely gone.

Besides, once you have re-usable building blocks, parametrizing them is the obvious next step. That's how libraries work.

[hunterloftis]

Then follow a common, well-understood example that has certainly not become turing complete:

https://developer.mozilla.org/en-US/docs/Web/CSS/@import

[ryedog]

+1 on include (or even variables that can be declared on the command line on build would be helpful)

[ChristianKniep]

+1 on include as well

I have to deal with one site where I have to set http_proxy and one without.

[kennu]

My +1 use case for INCLUDE would be to be able to offer third party components that are easily added to any existing Dockerfile. So if you want your Docker image to utilize a software product or SaaS service, you just add:

INCLUDE https://example.com/dockerfile

And it's ready to go. No need to add complicated preprocessing steps.

[dougmoscrop]

+1

I don't think INCLUDE is an affront to what Docker is trying to do at all. The fact that there exist containers that are all FROM ubuntu and install separate things like Ruby, Java, Node, and I have to create a new one that inherits from one, and copy-pastes from the others, just to have those three items available, says something is missing to me.

It would be great if INCLUDE fetched the Dockerfile from the registry that was used to build the referred image (making the registry more powerful), and I think a way of keeping INCLUDE in check might be that when a FROM is encountered in the context of an include, it asserts that the image being built shares a descendant with the included image.

So I can INCLUDE foo/ubuntu-java in my image that is FROM ubuntu but not one FROM centos

[prologic]

On Fri, May 30, 2014 at 12:26 PM, Doug Moscrop [email protected]
wrote:

It would be great if INCLUDE fetched the Dockerfile from the registry
that was used to build the referred image (making the registry more
powerful), and I think a way of keeping INCLUDE in check might be that
when a FROM is encountered in the context of an include, it asserts that
the image being built shares a descendant with the included image.

So I can INCLUDE foo/ubuntu-java in my image that is FROM ubuntu but not
one FROM centos

So this makes the proposed "INCLUDE" feature dependent on images and what
they're based on.
i.e: there's a direct relationship between FROM and INCLUDE where parent
base images have to match in some way.

Doesn't this complicate the implementation and usage a bit?

cheers
James

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

[dougmoscrop]

Could be.. I viewed it as a restriction that might mean more predictability for the result (amid concerns in another thread about 'grafting centos on to ubuntu'). My gripe is looking at the index and seeing a bunch of different things that are useful, and not wanting to just copy and paste their innards. You're right that there's probably (definitely) some hidden gotchas in what I'm suggesting.

I could rephrase my comments to be:

"As a Dockerfile author, I would like to re-use the work other people have put in to their Dockerfiles so that my specific instructions are more unique, and I'm willing to accept some kind of inheritance constraint when doing so".

[dougmoscrop]

I'll also add that, without some kind of mixin/include functionality, it's much more difficult to have - or rather, use - "authoritative" or canonical instructions (vs. "trusted builds"):

Imagine joyent/ubuntu-nodejs as a starting place. Now there's also oracle/ubuntu-java. If I want both, I have to create dmoscrop/ubuntu-nodejs-java and inherit from one while adding instructions copied from the other. Now someone else wants Java, Node and Ruby. Inheriting from mine feels "dirty" (because Oracle could easily push new updates to the instructions for installing Java and I've abandoned mine, or simply that image is stale) so they end up inheriting from one and copy-paste in the other two, and it goes on, combinatorially, as different people choose their inherit vs. copy sources.

Really, I don't care about dmoscrop/ubuntu-nodejs-java conceptually or as an artifact. I care about dmoscrop/my-project-builder which is simply FROM ubuntu INCLUDE joyent/nodejs INCLUDE oracle/java and some build steps.

[0xcaff]

@dmoscrop +1

[gzankevich]

+1 on include

[richardgill]

@dmoscrop +1

[ishanagr]

+1 this would be very helpful

[dergachev]

+1

[ChristianKniep]

To emphasis my usecase (a common one, I guess):

INCLUDE env.dockerfile

On-site:

$ cat env.dockerfile
ENV http_proxy http://theproxy.local:3128
$

Normal case:

$ cat env.dockerfile
$

Currently I have to differnet Dockerfiles.

[kennu]

@ChristianKniep I think your use case might be better served with a docker build -e option to specify environment variables when building an image. I myself could use both that and INCLUDE.

[ChristianKniep]

@kennu To simple! Oh my god, sometimes I wonder how I get up by myself. :)

[avelino]

+1

[mitar]

What about allowing multiple FROM to combine things? Or maybe MIXIN? So instead of including Dockerfile, one could combine multiple images?

[erikh]

Just marking that after #7461 we can tackle this if desired

[tiborvass]

How is this overlapping with #7115 or #7149 ?

[zeroasterisk]

+1 - I would also like this

[thorsummoner]

:+1:

[felixrabe]

+1 (and also +1 for having a way to vote on issues on GitHub so I don't have to +1-spam issues) :)

[erikh]

We’re actively addressing this.

On Oct 2, 2014, at 11:48 AM, Felix Rabe [email protected] wrote:

+1 (and also +1 for having a way to vote on issues on GitHub so I don't have to +1-spam issues) :)

—
Reply to this email directly or view it on GitHub.

[jokeyrhyme]

INCLUDE or IMPORT would greatly reduce duplication for me, as I have a project with multiple Dockerfiles, with various common tasks between them.

The only behavioural changes should be at the time of parsing the Dockerfile (by replacing the INCLUDE ... with the appropriate entries from that resource).

It may be useful to allow INCLUDE to work with HTTPS URLs, but I would still be very happy if it initially only supported local resources that are already able to be accessed with an ADD.

I actually just contemplated a build process to concatenate my snippets together into the final Dockerfiles I need, so I'm relieved to see that others have the same issues and that progress is being made. :)

[marcellodesales]

I have a runtime tha depends on:

• Ruby sources: some older scripts that were not ported to Node.js
• Node.js Sources: new micro-services that makes use of ruby services.

Right now I started a Dockerfile with Ruby, and then installed Node.js...

FROM binaryphile/ruby:2.1.2
MAINTAINER [email protected]

# Install Node.js and npm
RUN apt-get install -y python-software-properties --fix-missing && \
    apt-add-repository ppa:chris-lea/node.js && \
    apt-get update && apt-get install -y nodejs

Would this cover my requirement? I'm looking for something in the lines of:

FROM ubuntu/ruby:2.1.2
FROM ubuntu/nodejs 
...
...

Thanks!

[pypeng]

+1

[jokeyrhyme]

So, I'm not making any promises as I'm fairly new to Go, but I thought I'd try to tackle this over a few weekends. My progress will be found at https://github.com/jokeyrhyme/docker/tree/735-include-dockerfile until I'm ready to make a Pull Request.

Proposed specification:

• new IMPORT keyword for Dockerfiles (inspired by HTML and CSS, rather than PHP's include) with similar syntax compared to ADD, e.g. IMPORT path/to/fragment.to.read.in
• IMPORT sources must obey all the same rules as ADD sources, although initially I'll focus on individual source files (not archives, directories or remote URLs) unless it's super easy
• IMPORT lines can be anywhere in the Dockerfile
• there are no mandatory directives in individual IMPORTed source (e.g. FROM)
• IMPORTs can be chained when sources include more IMPORTs
• as this is a feature intended to maximise reuse, it makes sense to allow the same source to be
  IMPORTed multiple times whilst processing the same Dockerfile
• the final result should not break any Dockerfile rules
  
  
  
  • FROM still needs to be at the top, and there can only be one
  
  
• parser errors tell you which IMPORTed file is involved, if any
• the directed graph of IMPORTs should be acyclic (no circular references)

Typing this all out makes me realise that this is not a trivial issue. Any comments? Is this a fairly complete specification for how this feature needs to work?

[prologic]

@jokeyrhyme What does an IMPORt look like in a Dockerfile?

[jokeyrhyme]

@therealprologic e.g. IMPORT path/to/fragment.to.read.in

[prologic]

Ahh gotcha :)

[suntong]

+1.

Please review jokeyrhyme's patch and incorporate it soon.

I am using Docker to do my Debian/Ubuntu Package Developing, the only difference between my Debian DockerFile and Ubuntu's is their FROM line. All the rest are the same! So you can see how that INCLUDE is useful to me.

PS. I'd rather it is named as INCLUDE, not IMPORT, following C/C++. IMPORT implies more action, e.g., it's a perfect word for specifying multiple images per context, if it is ever possible.

[cc272309126]

I want to know if the “import” feature of dockerfile can be used now？ I have write a ugly version of this feature。https://github.com/cc272309126/docker/commit/3f7e775e3979cb0cff9df84c1d75d1d2d14b3348

[cc272309126]

the test case I have not finish it. it is not for import but for include. that means you can use include /dockerfile in your Dockerfile

[jokeyrhyme]

That's great, @cc272309126. My own work on this is slow. I'm having trouble running the Docker automated tests on my system (OSX with Boot2Docker), and I haven't had much time to work on this.

[analytically]

+1

[cc272309126]

Hi,I have a question about the import command. like if dockerfile B want import A from a url like http:xxx/xxx/A. and A has ADD and COPY command in the dockerfile. How I get the ADD and COPY command's source files？ one way is I should prepare the source files advance。 but for import command I think it should not care for the source files。the another way is that I should write the ADD command in dockerfile A like ADD http:xxx/xxx/files dir。but when I write the dockerfile。I should not care for which url I should put the dockerfile。

[jokeyrhyme]

@cc272309126 is it okay if all ADD and COPY directives (even in IMPORTed files) work from the directory of the root Dockerfile as they normally would? The end result is just as though you only had 1 big Dockerfile?

[cc272309126]

@jokeyrhyme yes, it is okey for all ADD and COPY directives (even in IMPORTed files) work from the directory of the root Dockerfile. the end result is not a big Dockerfile. it build the imported dockerfile when it run into the IMPORT command. And all the resource of imported dockerfile is relative from root Dockerfile's path。

[ahoy-jon]

I did a bash script to compensate the missing INCLUDE : https://gist.github.com/ahoy-jon/e8c534883a5a8c87c046, but finally I use it to compose existing Dockerfile (removing FROM directives from sub Dockerfile). It doesn't support proper ADD (eg : an ADD from an included URL should resolve on the network, relatively (an ADD from an included file should resolve relatively too ? )).

[suntong]

I did a bash script to compensate the missing INCLUDE ...

A showcase of how badly that people need this, every one is jumping through the hoops in their own way trying to fix the problem. ...

• cpp can do recursive but won't fetch from the web
• this one can fetch from the web but can't do recursive

we need a solution, not patches on patches.

[ahoy-jon]

This bash solution is recursive, but doesn't solve stuff around ADD.

I changed now the example to make it clear. Here is the recursive part of the script : https://gist.github.com/ahoy-jon/e8c534883a5a8c87c046#file-m5-bash-L31

[suntong]

On Sat, Dec 27, 2014 at 5:05 PM, Jonathan Winandy [email protected]
wrote:

This bash solution is recursive, but doesn't solve stuff around ADD.

Oh, yes. Sorry.

[deni64k]

That's will be great! :+1:

[ajvb]

This would be really nice, especially when combined with the docker build -f flag that has recently been merged into master. Allows for multiple dockerfiles with a core one so that they can stay DRY.

[mpareja]

-1 Include would only solve a very narrow use-case. You'll soon find yourselves needing just a slight bit more power.

This is why tools like chef, ansible and puppet have thriving ecosystems of "cookbooks/recipes", "playbooks/tasks" and "resources". They also allow you to tweak how those things are applied. While I don't want to manage everything using these tools, I do appreciate the re-usability and composability you get from a library of tools to help you get stuff on a machine - even if it is just once during a docker build.

Disclaimer, I haven't tried using something like chef _inside_ the image build process, but I've certainly felt the pain of wanting to add the same functionality to multiple images and copying it around in Dockerfiles. I also don't think using parent images to DRY (don't repeat yourself) up your Dockerfiles is a good idea. Parent images remind me a lot of inheritance in object oriented programming - it's either precisely what you need or steer the hell clear of it because the coupling makes changing things very difficult.

[duglin]

@tiborvass @jfrazelle @crosbymichael
I'd like to hear your take on this as the topic of an INCLUDE does pop up from time to time.

Personally, I don't see an issue with it since adding it doesn't break any design principle of Docker and if it makes people's lives better (or at least if they think it does :-) ), and there is no nice alternative, then I'm ok with it. I don't really see it being much different than an "include" statement in a Makefile.

I would however suggest that we allow for it to be a bit more powerful by allowing env vars to be used - e.g.:

INCLUDE dockerfiles/os-specific.${os:-default}

But we can always do that as a secondary step.

(edit: added bit about no nice alternative)

[madd512]

If you want to generate Dockerfiles with includes, conditionals, and turing completeness you can use dockerscript. Full disclosure: I made this because I wanted includes.

[jessfraz]

Hello!
We are no longer accepting patches to the Dockerfile syntax as you can read about here: https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

Mainly:

Allowing the Builder to be implemented as a separate utility consuming the Engine's API will open the door for many possibilities, such as offering alternate syntaxes or DSL for existing languages without cluttering the Engine's codebase

Then from there, patches/features like this can be re-thought. Hope you can understand.

[jcoffland]

I know you guys are stubborn as hell on this topic, which is commendable, but why doesn't docker just run m4 on Dockerfiles by default. This would give docker many powerful features including include() with out changing the syntax. docker build would need to be modified to accept some m4 command line options such as -I but that's about it.

[0xcaff]

Its because we want an include directive which is smart and merges the images together. Someone correct me if I am wrong. Saying use m4 is like telling a car owner to start their car with a screwdriver. It works but its not elegant and adds more dependencies.

On Wed, Oct 21, 2015 at 1:26 PM -0700, "Joseph Coffland" [email protected] wrote:
I know you guys are stubborn as hell on this topic, which is commendable, but why doesn't docker just run m4 on Dockerfiles by default. This would give docker many powerful features including include() with out changing the syntax. docker build would need to be modified to accept some m4 command line options such as -I but that's about it.

---

Reply to this email directly or view it on GitHub:
https://github.com/docker/docker/issues/735#issuecomment-150014366

[thaJeztah]

The reason this feature is closed now, is that we are working on moving the builder out of the daemon to the client. This allows other implementations (you'd even be able to create your own Dockerfile alternative), see https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

Before that has arrived, we don't want to make changes to the Dockerfile syntax. Apart from that, we want to keep the Dockerfile syntax simple (we will never be able to have a one-size-fits-all solution), hence allowing other implementations that can be tailored more to specific needs.

[jcoffland]

@caffinatedmonkey I think you are assuming that m4 is not good because it's old. m4 is more like a swiss army knife than a screwdriver. But what does this have to do with starting cars?

@thaJeztah "we don't want to do it now" is a terrible reason for closing an issue. This is exactly why you want to leave issues open.

[thaJeztah]

@jcoffland it was decided to close these issues until it's become more clear what the future of the builder/Dockerfiles will look like, but please "ping" once the builder change is implemented.

I'll also ask among the other maintainers if re-opening these requests before that is something to consider

[jcoffland]

I recommend using GitHub milestones. You could reopen these issues and put them in a future milestone. In my experience, this keeps issue reporting users happy because they don't see you dismissing ideas they consider important. Yet you can still easily sort out issues which are not currently important to you. I often create a non-versioned milestone with a name like unscheduled for this purpose.

[0xcaff]

@jcoffland I'm not assuming m4 is not good because it's old. It is an example of design has stood the test of time. I was trying to use the car screwdriver thing as a metaphor.

[tomalakgeretkal]

I would have used this so that my 32-bit* and [native] 64-bit CentOS 6 images wouldn't be 184 identical lines with a couple added to the top of just one.

* Same approach as https://hub.docker.com/r/toopher/centos-i386/~/dockerfile/

[jcoffland]

@tomalakgeretkal I use m4 for exactly this purpose. You can do so now by preprocessing your Dockerfile with something like this:

m4 -I . base.m4 > Dockerfile

It would be even better if docker either had an option to automatically preprocess with m4 or better yet always did so. Docker would need to add -I, for include, to the docker build command but that's about it. Supper easy to implement and a general solution which solves a lot of problems.

[rncry]

+1 for this (whenever it gets reopened)

[assalw]

+1 This would really be great! I need this to structure my Docker files in a sensible way.

[kujiy]

+1
WE REALLY NEED!

[avirshup]

Docker Make might be a solution to many of the problems here. It supports both:

1. multiple dockerfile "inheritance", and
2. ADDs/COPYs from multiple locations.

Here's @binocarlos 's example, with files added from multiple locations. You specify the build with a YAML file:

imagemagick:
 build_directory: /path/to/data
 build: |
   RUN apt-get install imagemagick
   ADD x.png #adds /path/to/data/x.png

redis-session:
 build_directory: /another/path/
 build: |
   RUN apt-get install redis-server
   ADD y.dat #adds /another/path/y.dat

combined:
 FROM: appserver
 requires:
  - redis-session
  - imagemagick

And create the final image just by running

docker-make.py combined

Full disclosure: I wrote DockerMake because I was sick of dealing with "FROM" dependencies.

[kujiy]

@avirshup Is Docker-Make a tool like docker-compose for making a docker image?

[juanmirocks]

+1

[jakirkham]

Yeah, the idea of parsing Dockerfiles through m4 is abhorrent to me for exactly the reason mentioned by @caffinatedmonkey. Namely we are shipping off a feature that really belongs in the core spec. This means it will be reimplemented in various different ways and I may not be able to reuse the same strategy that someone else has as there is no official standard (worse code duplication than just setting and enforcing a standard). Not to mention, we have now added a dependency that will be hard to get on some platforms. As far as a path, we are already looking at paths for context anyways and we already have mechanisms for changing them. In short, I prefer the original proposal.

[Vanuan]

Another solution is Rocker: https://github.com/grammarly/rocker
IMO, it solves all the weak points of Dockerfile, making it suitable for development.

[rgpublic]

Rocker doesn't seem to have INCLUDE. At least I can't seem to get it to work. Surprisingly complex issue, indeed! I'm a novice Docker user and was really suprised that sth. so simple and straight-forward like an INCLUDE statement isn't - no pun intended - included. If Rocker doesn't have this as well I think I will need to write some kind of custom script to merge multiple files together, Oh, well...

[Vanuan]

@rgpublic it's called IMPORT

[rgpublic]

@Vanuan: Thank you so much for your response, but I think IMPORT does sth. different:

INFO[0000] IMPORT "includes/init.rocker"
FATA[0000] You have to EXPORT something first in order to IMPORT

I don't want to export sth. first or move stuff between images, I originally only wanted to split a long and confusing .docker file into multiple ones to make it easier to read and more modular. Just like the INCLUDE statement in PHP, CSS, C++ and many other languages. Isn't that what INCLUDE is all about? Strangely, they even added INCLUDE to the syntax highlighting even if it doesnt work:

https://github.com/grammarly/rocker/issues/29

This all simply turns into too much of a mess for me for such a "minor" issue. I think I'll simply write a small preprocessor script to sort this out.

[Vanuan]

Yeah, it's a different one. Sorry

[phpdude]

+1 to INCLUDE

[adw0rd]

:+1:

[boneyao]

+1

[0xcaff]

😭 Does no one realize the +1 spam isn't necessary anymore? https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments

[igorpejic]

+1

[fengli79]

INCLUDE +1

[bitsofinfo]

+1

[nfedyk]

+1

[peloncano]

+1

[TalBeno]

It is mind boggling that there is no way to reuse building blocks i.e. collection(s) of layers per functional intent, at arbitrary positions within a dockerfile. With INCLUDE or any equivalent solution. What am I missing?

[Vanuan]

The thing I was missing previously is that each line in Dockerfile creates a new layer. Thus you should never have more than 5-10 lines in Dockerfile to keep image minimal.

[suntong]

Thus you should never have more than 5-10 lines in Dockerfile to keep image minimal.

Not necessary. I put in whatever necessary, then remove the intermediate containers/layers to save space & speed things up with the -rm switch

[thaJeztah]

@suntong the --rm flag only removes the intermediate container, but does nothing to optimize your image, also there's no need to specify it, as it's actually the default:

 --rm                      Remove intermediate containers after a successful build (default true)

Intermediate _layers_ will still be added, so if (for example) you ADD a 100 megabyte file, and in the next step RUN rm big-file, your final image will still be 100 megabyte in size.

However, that discussion is separate from the discussion at hand here; an INCLUDE statement would only be for _convenience_ and not make any difference other than that.

[bmudda]

+1

[Vanuan]

Yeah, it would be much more convenient to create thousands of layers and monolithic apps. Personally, I wouldn't want to see such Dockerfiles in the docker ecosystem.

Let's look a the use case.

It sounds like people want multiple inheritance here, to build monolithic images which use multiple languages and technologies.

AFAIK, the philosophy of docker ecosystem is to break down monolithic apps into distinct microservices by technology. So that you wouldn't need to import ruby Dockerfile into nodejs Dockerfile.

That what I believe is the rationale behind the hesitance to include INCLUDE: to enforce microservices architecture to keep Docker ecosystem a lightweight look and feel.

[suntong]

On Wed, Oct 12, 2016 at 11:27 AM, John Yani wrote:

It sounds like people want multiple inheritance here, to build a
_monolithic_ image which uses _multiple languages and technologies_.

I don't agree. But anyway...

after all, "restricting how people use software" is what MS does best,
and what I dislike.

I believe, in essence, docker is meant to give people freedom,
not restriction.

[leethomas]

Since it doesn't look like this will ever be officially supported, I wrote a command line tool called harbor that suffices for my personal use cases.

While it supports recursive INCLUDEs, I've only tested it in moderation for my purposes. Haven't tried loading 1000+ files or anything. Posting it here in case anyone else might find it useful.

Obviously some directives should not be included multiple times in a single Dockerfile. Like FROM or CMD for example, so take caution there.

[artheus]

Edit: Removing my +1
I no longer see the need for a feature like this in Dockerfiles.

[josephtyler]

+1000

[vito-c]

@jfinkhaeuser

Your suggestion is flawed in several ways:

1. It doesn't take advantage of image caching.
2. It adds two FROM statements and two CMD statements to the output
3. /* is used in the *.docker files as a path expressions (ie rm -rf /var/lib/apt/lists/*) but is interpreted as a comment and has to be escaped /\* (most dockerfiles don't do this).
4. A comment in a Dockerfile starts with # but is a preprocessing directive so you have to strip all of your Dockerfiles of comments.

I do see your point about a preprocessor though and I also think that INCLUDE is a bad idea.

Original Suggestion:
Makefile:

Dockerfile: Dockerfile.in *.docker
    cpp -o Dockerfile Dockerfile.in

build: Dockerfile
    docker build -rm -t my/image .

Dockerfile.in:

FROM ubuntu:latest
MAINTAINER me

#include "imagemagick.docker"
#include "redis.docker"

imagemagick.docker: https://hub.docker.com/r/acleancoder/imagemagick-full/~/dockerfile/
redis.docker: https://github.com/docker-library/redis/blob/6cb8a8015f126e2a7251c5d011b86b657e9febd6/3.0/Dockerfile

[jfinkhaeuser]

@vito-c

1. I don't see your first point. The generated Dockerfile would be the same if none of its input files change, which means image caching works just as well with as without generating the file. Maybe I misunderstand something?
2. It only adds FROM and CMD multiple times if you intend to put them into the included files. I don't think that's good form for something that's essentially a fragment of a Dockerfile. An INCLUDE statement would not change that unless it explicitly strips these from the included file.
3. So quote paths. That's good form anyway because of variable expansion and spaces: RUN rm -rf $SOME_PATH/lists/* might not do what you want if $SOME_PATH contains / /tmp. Quotes are your friends with anything related to shell scripts and variables.
4. So use /* */ for comments.

Really, these arguments make sense if you expect to just #include any downloaded third-party Dockerfile. But if you're building your own setup, you just have to adapt to the choices you've made. If your choice is to use the C preprocessor, then these are the consequences.

[bobbui]

+1

[tetherit]

Is this available already?
EDIT: In that case +1 :)

[duglin]

@xanview nope

[Vanuan]

@xanview it is currently recommended to use FROM approach + multi-stage builds:
https://github.com/moby/moby/issues/2745#issuecomment-319839656

[rulai-huajunzeng]

Multiple stage build is only a workaround and it's not the best workaround in my opinion. It brings additional burden of maintaining intermediate images and their versions and you can easily forget to rebuild those images when the codes are updated.

The best workaround till now is still cp someLib . && docker build . && rm someLib. Or similarly you can use six8's dockerfactory. Unfortunately these don't work when you want to do docker-compose build.

[Vanuan]

you can easily forget to rebuild those images when the codes are updated.

similarly, you can easily forget to cp someLib

[tetherit]

@huajunzeng What do you mean by cp someLib && docker build . && rm someLib? - why would you do that? -- couldn't the docker file wget it or something, how does this address splitting a Dockerfile into multiple smaller chunks?

[rulai-huajunzeng]

@xanview My understanding of the whole point of this discussion is all about multiple build context. Whether to use INCLUDE, or to COPY from parent directory, etc. By using cp we can just manually pack different directory into a single build context and send to docker.

[Vanuan]

is all about multiple build context

No, I think it's about reusing multiple Dockerfiles. E.g. when you want to have nginx and node in the same image.

By using cp we can just manually pack different directory into a single build context and send to docker.

That's the thing, you can't copy something from an image. Consider you have nginx image:

FROM nginx

ADD nginx.conf /etc/nginx/nginx.conf

If you want to install node in addition to nginx:

FROM node

ADD . /src/
WORK_DIR /src/
RUN yarn install

Copy command doesn't help you here anyhow. Only if you use multistage builds you can copy files between images:

FROM nginx

FROM node
ADD . /src/
WORK_DIR /src/
RUN yarn install

COPY --from=0 /path/to/nginx/resources /path/to/nginx/resources
ADD nginx.conf /etc/nginx/nginx.conf

I agree that this doesn't cover other use cases like exposed ports, entrypoints (how would you combine them?), users (tricky), etc. And you also have to know which folders contain needed files and libraries. Base images also need to be compatible (library versions, link symbols, header locations). But it's a start.

---

What's people really intended to do with IMPORT is this:

Dockerfile1:

FROM ubuntu
RUN apt-get install nginx

Dockerfile2:

FROM debian
RUN apt-get install node

IMPORT Dockerfile1 # => RUN apt-get install nginx

But that's kind of counter-intuitive. The better way is to use some bash script ./install-nginx.sh and copy it where needed. That's probably what @huajunzeng meant.

[zh794390558]

Does this feature can be accesed now?

[davesque]

Here's a use case for INCLUDE:

Heroku's support for docker containers allows one to push multiple images for different process types in this manner:
https://devcenter.heroku.com/articles/container-registry-and-runtime#pushing-multiple-images

Say you have a Django app which defines some asynchronous Celery tasks. Then you need a web process type and a worker process type. But they should both have essentially the same runtime environment. It would be convenient to just have your base Dockerfile.web which looks something like this:

FROM python:3.6

...

CMD ["web-entrypoint.sh"]

and then have Dockerfile.worker look like this:

INCLUDE Dockerfile.web

CMD ["worker-entrypoint.sh"]

As it stands, Heroku provides no way to specify a different command using the same image for a different process type. And Docker provides no way to do simple Dockerfile templating. So I'm stuck with duplicating the same Dockerfile and changing only one line.

[duglin]

@davesque you may want to continue to chat over here: https://github.com/moby/moby/pull/12749

[Stannieman]

I also felt the need for this for some multi stage dockerfiles where some of the stages are exactly the same for several dockerfiles. I know there are other ways to solve this, for example building the stages as separate images, but that's all a bit overkill for a relatively "simple" environment.

I ended up writing the PowerShell script attached.

IncludePreprocessor.txt

Call WritePreproccessedFile with the dockerfile to process and the output file as parameters.
It's fairly simple and maybe not super robust, but works perfect for me.

To include a file to your dockerfile do this:
INCLUDE path/to/../../other/file with spaces

This is recursive so you can have other includes in the included file.

[himslm01]

My use case is for reusing a block of ARGs with default values in multiple places.

I have one Dockerfile which downloads the source and libraries for a project. That Dockerfile needs to know the versions of the libraries to download.

I have a second Dockerfile for building the source into an executable. I build from the second Dockerfile multiple times to build for various architectures, having downloaded the source only once. The second Dockerfile needs all of the ARGs of the first Dockerfile for the versions of libraries, plus extra args for build parameters for the different architectures.

I can't even use a multi-stage build, because the ARGs are reset after every FROM, so even in a multi-stage build I do not have one-single-source-of-truth, I have to have the ARGs defined twice in one single file.

This situation is not architecturally right!

[rodvdka]

+1

[bg172]

+1

[kkm000]

I am mildly missing this feature for testing multiple platfrorms. This would minimize maintenance, I absolutely do not want to copy/paste common stuff. Currently, bash and cat is my preprocessor. Not super important, but would be nice to have. Nothing Turing-complete here:

FROM opensuse/amd64:13.2
INCLUDE prereq-suse.inc
INCLUDE common.inc

This pattern (if INCLUDE were a thing) would work in my case for opensuse, debian, ubuntu, centos and partly Fedora ("old" Fedora/"new" Fedora single split).

I absolutely do not buy the argument that adding the INCLUDE statement to your declarative language inevitably leads to Turing-completeness, obesity, fleas on your dog and crime in your neighborhood. This is a stretch far and thin.

[borgogelli]

+1

[anton-matosov]

Tensorflow team have implemented this functionality using an assembler.py python script
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/dockerfiles which assembles partial dockerfiles and the rules in spec.yml

Edit:
Here is an RFC describing their approach in details https://github.com/tensorflow/community/blob/a8f3baec6c90197d697796004d6352fc942c4bf1/rfcs/20180731-dockerfile-assembler.md

[hholst80]

It might not be obvious to everyone how to use the multi stage builds to work around the missing INCLUDE. The example below should be fairly self explanatory. Another obvious thing is the inefficiency that the staging build steps are always done even for "production" images.

See also https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact

ARG env=production

FROM nginx:alpine AS base

FROM base AS production
COPY nginx.conf /etc/nginx/nginx.conf

FROM base AS staging
COPY nginx.no-ssl.conf /etc/nginx/nginx.conf

FROM $env

[thaJeztah]

Another obvious thing is the inefficiency that the staging build steps are always done even for "production" images.

Try setting DOCKER_BUILDKIT=1, because that's only the case when using the classic builder; when using buildkit, those stages will be skipped if they're not needed to produce the final stage

[zh794390558]

+1

[MBeijer]

+1

[vrothberg]

Buildah supports including other Dockerfiles since 2018. To avoid extending the Dockerfile syntax and remain compaitble, it's using the C pre-processor to assemble the Dockerfile (see the Buildah docs). Buildah only runs cpp on a Dockerfile if it's suffix is .in. Other Dockerfiles can be included via the #include $PATH directive.

Certainly, the C pre-processor allows writing macros and we can do horrible things with it, but it's used and understood since the 60s and present on pretty much all systems, giving a certain degree of portability. Note that this method does not require Buildah at all and it can be used by pretty much everybody trying to implement include-semantics for Dockerfiles. Simply run cpp -E -iquote . Dockerfile.in and the output can be parsed by any Dockerfile-compatible tool.

[tanzislam]

If you're using the preprocessor in the latest versions of GCC for this purpose, then you'd need the following switches:

• -traditional to prevent GCC from removing trailing backslashes in non-directive lines.
• -undef to prevent replacing any occurrences of the word "linux" in your Dockerfile with "1", etc.

The cpp preprocessor in Xcode Command-line Tools also accepts these switches without any behavior changes.

[ringerc]

Spell it as FROM PATH if you're unhappy with calling it "include". But please, please consider adding this, the lack is painful and forces higher levels of a stack to do direct Dockerfile munging when they should be able to consider a single Dockerfile + context dir to be an authoratative source.

[dougmoscrop]

I still think this could be accomplished in a way that is (hopefully) compatible with the current APIs as long as the containers shared an ancestry, i.e. someone should be able to do

FROM alpine
INCLUDE alpine/xyz
INCLUDE alpine/abc
blah blah

at that point the alpine/xyz just becomes a way of factoring out commons steps. could xyz install something and abc needs a different version? of course, but that's no different than building a container that has two separate package install steps - in fact, it becomes that, it's just about DRYing the canonical build steps for e.g. java, node, etc.

[klakegg]

This is a crucial feature when moving forward with multiple stages/targets/architectures, and has finally grown into an itch I simply had to scratch.

My humble solution is preprocessing of the Dockerfile so a few lines of code may do some imports and update some references in the imported instructions. It is available here:
https://github.com/klakegg/dockerfile-import

This is not a perfect solution, however I guess it may relieve some of the pain until it is fixed in Docker.

[tantal92]

another scenario is when you have many images, and in every image you need to set up the same set of ENV variables;
(e.g.: INCLUDE file_with_my_envs)