Docker-maven-plugin: Support for docker-compose

[rhuss]

Add support for Fig and the successor Docker compositions via the new external configuration mechanism.

[mwl]

Docker compositions looks like the one to go for in my opinion?

Another feature to consider in that regard is producing a Docker composition group.yml file as an Maven artifact.

[rhuss]

Yes, I also think to fully support group.yml but since it's not yet available and fig.yml seems to be quite close in syntax, supporting both simultanously shouldn't be that difficult.

Creating a group.yml for the other way round is a great idea ! Just have to think a bit about how to include that (probably as you said as a new artifact type, but maybe there are other options, too.)

[mwl]

The use case I have would be something like

1. Maven module produces the usual fat-jar and builds a Docker image around it using build goal
   
   
   
   • CI deploy configuration will also push it to the local Docker repository
   
   
2. Maven module depends on previous module and produces the group.yml file
   
   
   
   • CI deploy build will publish the group.yml file in the local Maven repository
   
   
3. Maven module for integration test spins up the group.yml produced before and runs Failsafe tests against it.

I believe the approach is in the spirit of both Maven and Docker?

[rossbachp]

I have successfully tested docker-compose release 1.1.0-rc2 https://github.com/docker/fig/releases/tag/1.1.0-rc2. Good time to start implementation....How I can help?

[jgangemi]

has any thought been given on how this type of configuration could be exported into a maven repository so it could be imported into another pom? i thought i saw something about that floating around but i can't seem to find it now.

[benjamindonze]

Has some work been done here? I would be very interested in this feature. Plugin beeing able to start container based on docker-compose.yml file. This way I wouldn't have to keep updated both my pom.xml for my automated integration tests and a docker-compose.yml file for my productive environment.

[jgangemi]

no - nothing by us to my knowledge at this point. some where there is another issue floating around where someone stated they had an implementation of this in a forked project. we were hoping they'd contribute it back (even if it was rough around the edges) but i'm not sure where that conversation left off.

i'll see if i can dig up the ticket, if @rhuss doesn't find it first, and reach back out.

[jgangemi]

oh - duh - said issue is #273 referenced above.

[jgangemi]

i'm going to pick this up, there is already support for this using properties files, so all that should be needed to provide a basic level of support is a yaml parser.

[jgangemi]

so i figured i should drop this here for feedback before i potentially get to far...

adding compose support will be somewhat of a small breaking change to the xml which will only really affect ppl who are using the external property stuff.

here's what the new config looks like (more or less)

      <plugin>
        <groupId>org.jolokia</groupId>
        <artifactId>docker-maven-plugin</artifactId>
        <configuration>
          <images>
            <image>
              <external>
                <compose>
                  <dockerComposeDir>kafka-docker</dockerComposeDir>
                  <extended>
                    <service>
                      <id>zookeeper</id>
                      <wait>
                        <time>1000</time>
                      </wait>
                    </service>
                    <service>
                      <id>kafka</id>
                      <wait>
                        <time>1000</time>
                      </wait>
                    </service>
                  </extended>
                </compose>
              </external>
            </image>
            <image>
              <external>
                <properties>
                  <prefix></prefix>
                  <propertyFile></propertyFile>
                </properties>
              </external>
            </image>
            <image>
              <external>
                <other>
                  <type>foo</type>
                  <configuration>
                    <name>value</name>
                  </configuration>
                </other>
              </external>
            </image>
          </images>
        </configuration>
      </plugin>

external properties have been moved to their own configuration section (w/ a future enhancement to allow those props to live in a file) and anyone can still add a handler of their own choosing using the other block.

since docker-compose doesn't support some of the advanced features that the plugin offers, the wait, log, and watch configuration elements will be made available via the extended block and keyed to the service in the compose file using the corresponding id (w/ a future enhancement to also allow these values to live in an external property file).

also - the first cut of this support will not support building containers. there are some open questions regarding how best to support that, some of which is slated to be supported in the docker-compose.yml file itself. advanced features such as scaling, etc also won't be available to start either.

[rhuss]

sounds good to me. I like the idea to use dedicated tags for other kinds of configuration so that we could have a better XSD support for IDE completion (which I want to add soon).

Frankly, I'm not so keen on the external property configuration since it adds quite some maintenance burden to not forget to new config options there, too.

Maybe we can get even rid of the <external> and define instead a the alternative configuration approaches directly. Of course we need some validation which combinations of <build>, <run>, <properties>, <compose> is allowed with which priority.

Some random remarks:

• would call it directory instead of dockerComposeDir since _compose_ is clear from the context. Also, why not pointing directly to the compose file ? Sounds more natural to me
• Instead of <extended> ---> <services> as list container for <service> elements. Its also less about the technical detail (that this sections extends the compose stuff) and more about what it adds.

[jgangemi]

i don't think the external section is horrible to have. it gives some meaning that this configuration is coming from someplace other then the direct xml configuration. i agree that the additional forms of specifying a config means more work but i'm not sure there's really any way around that.

part of me thinks we should ditch the configuration that supports an arbitrary map b/c are ppl really using some other external format then docker-compose? at the same time, it's not hurting anything leaving it there. i did, however, discover that the plexus wiring is slightly incorrect and if anyone was trying to create a handler as a separate plugin, it wouldn't work (we need to define role-hints for the components otherwise plexus chooses the first one it sees as default and all others get ignored) and we'd have gotten a bug about it.

as to your remarks...

i can change extended to services, that's easy and i'm always open to better naming. same for the compose directory. i do want to support building images (and i realized we can just use the service config to let the user choose an image name) and i went w/ directory b/c that's what we did for the Dockerfile in the assembly. i am going to add a config element to to specify an alternative filename other then docker-compose.yml. it also means i don't have to do any extra parsing if the build dir is specified as ..

p.s. i did not forget about #87, that's been getting love during work day hours. should have something for that in the next couple of days.

[jgangemi]

fyi: i'm about 75% of the way through the run portion of this. i got caught up w/ some other things and am trying to get back to it. given this will be a 0.14.x feature, i think i'm ok on time. :)

[rhuss]

@jgangemi any news on this ? ;-)

[jgangemi]

unfortunately no - i have had no spare time as of late to get anything done. if i can't make any more progress on this by the end of the weekend i can throw up what i have in a branch and you can have at it.

[rhuss]

no problem, I know how this is ;-)

Feel free to commit ll you have in a branch, I would happily have a look at it and pick it up ...

[jmMeessen]

@jgangemi , @rhuss Would sending some Belgian Chocolates help to get this feature done ? I would happily do, This feature is waited for by several people over here.

[jgangemi]

@jmMeessen ooh, perhaps :)

are you looking to be able to build containers from compose configurations as well or just run them?

[jmMeessen]

Located in Brussels, I am near the supply ;-)

Currently, we are looking to just start (=run) containers. We use the integration test setup to load data or applications in the containers.

Being able to also build containers would be neat, but we could wait for that.
I can check it out/confrim with my colleagues (as I described my use case)

[jgangemi]

would it be possible for you to post one of your compose configurations here?

i had most of the run stuff working w/ the exception of reading in a couple fields where they decided to overload how the yaml works, i think related to environment variables and something else but i need to go back and check.

[jmMeessen]

I will do that tomorrow morning

[jmMeessen]

The colleague interested in the feature will post directly his use case here.

[jgangemi]

i have a beta version of this that you can try out, just use the following version in your pom:

0.14.0-compose-SNAPSHOT

there is support building and running containers, with the following exceptions:

• no extends support
• no env_file support. i have an idea for it, just not done
• no variable substitution support yet.
• anything not currently supported by the plugin (such as some run options, etc)

the plugin configuration now looks like this for the external configuration providers. gone is the ability to define a type of other. it's already difficult to support two external configuration providers and a direct integration is probably better then something adhock anyway.

      <plugin>
        <groupId>org.jolokia</groupId>
        <artifactId>docker-maven-plugin</artifactId>
        <configuration>
          <images>
            <image>
              <external>
                <properties>
                  <prefix>prefix</prefix>
                  <propertyFile>path/to/propertyFile</propertyFile>
                </properties>
              </external>
            </image>
            <image>
              <external>
                <compose>
                  <basedir>src/main/docker</basedir>
                  <yamlFile>kafka-docker/docker-compose.yml</yamlFile>
                  <services>
                    <service>
                      <image>issue-57-compose-test:${project.version}</image>
                      <alias>zookeeper</alias>
                      <build>
                        <cleanup>true</cleanup>
                        <nocache>false</nocache>
                        <compression>gzip</compression>                        
                        <skip>false</skip>
                      </build>
                      <run>
                        <portPropertyFile>/tmp/issue-57-compose-test.properties</portPropertyFile>
                        <wait>
                          <time>1000</time>
                        </wait>
                        <skip>false</skip>
                      </run>
                    </service>
                  </services>
                </compose>
              </external>
            </image>
          </images>
        </configuration>
      </plugin>

if you notice, the configuration likes almost identical to what you'd define in the pom w/ the exception of the services/service tags, and under the hood they are. by the time i added elements for the above options, it seemed better to just reuse the existing format. it also means that if we wanted to, we could allow overriding parts of the compose configuration via the pom.

basedir is not required and defaults to src/main/docker.

compose configurations which require a random port will automatically have a ${alias}_port_${position} property created and made available to them and also conforms to how the plugin currently works. when the property names are generated, ${position} will be equal to where that port is defined in the list definition, eg:

id: application
ports:
  - "8080:8080"
  - "8181"
  - "8282:8282"

will place the container port into a property named application_port_2 b/c it's the 2nd item in the list.

i still have to finish tests and documentation, but i'll throw up a PR to at least get a review going.

comments, feedback are welcome...

p.s. i tested this against wurstmeister/kafka-docker compose configuration.

[sebglon]

Is it released?
What is the new milestone?
My company expects this feature to advance to the prod

[rhuss]

sorry, not yet. But its next the roadmap along with Dockerfile enhancement. Initial work has been done by @jgangemi in https://github.com/fabric8io/docker-maven-plugin/pull/384, need to finish integration thouhgh.

Unfortunately I'm currently busy with other work until easter, but will continue work on this after this.

The new milestone will be 0.15.0 (since some major changes are expected), I will update the roadmap accordingly.

[jhovell]

Noticed it's been a few more months, is this being actively developed? Looks like a very useful feature, in particular if "extends" support is added as well https://docs.docker.com/compose/extends/#/extending-services

[rhuss]

There is work going on, unfortunately a bit slow. The changes are tracked in a dedicated branch https://github.com/fabric8io/docker-maven-plugin/tree/384-compose (the original PR is https://github.com/fabric8io/docker-maven-plugin/pull/384). So it will eventually end up in this plugin, not quite sure when (although we quite close). There is still some discussion how to add d-m-p specific feature configuration to either the compose file as extension or as part of the d-m-p xml configuration (as it is at the moment)

Thats however only for the direction compose -> d-m-p (i.e. a compose file as configuration for d-m-p). The work on implementing the creation of a compose file from d-m-p configuration has not yet started, though.

That said, docker compose's extend wont properly make it in to this first implementation (mainly because of lack of time), although I consider this as very useful, too.

If you are interested, we happily integrate contributions, too. If so, I could elaborate a bit was is still missing on #384 to make it into one of the next releases.

[AlexMaskovyak]

I'm investigating your docker-maven plugin for use in my projects but I'm a bit put off by the need to duplicate my compose file in the pom.xml. What's the status of this feature?

[jgangemi]

it works - what specifically do you find yourself needing to duplicate?

[AlexMaskovyak]

I had thought that this issue was still open. I have a compose file that that I use for running my service and linking it to dependent services. I'd like it if I didn't have to replicate this in my pom.xml for running tests.

[jgangemi]

you shouldn't need to duplicate anything in the pom to make it work.

https://dmp.fabric8.io/#docker-compose

[AlexMaskovyak]

Awesome, I looked at the open issues before I actually explored the documents. This looks like exactly what I need.

Out of curiosity, why is this issue still open?

[jgangemi]

no real reason other then poor house keeping - i think it can be closed.

[raahool]

Does docker-compose option works in detach mode ? I am not able to use it with detach mode!
Please refer to https://github.com/fabric8io/docker-maven-plugin/issues/1066