Nomad: Command not running when docker container has an ENTRYPOINT

[rickardrosen]

Nomad version

0.5.1

Operating system and Environment details

Ubuntu 16.10
Docker version 1.13.0, build 49bf474

Issue

A dockerfile with an ENTRYPOINT ["/run.sh"]
When I schedule it with:

              "command": "/local/containerpilot",
              "args": [
                "-config",
                "file:///local/containerpilot.json",
                "/run.sh"
              ]

docker ps shows COMMAND:
/run.sh /local/containerpilot -config file:///local/containerpilot.json /run.sh

I would expect the COMMAND to be:
/local/containerpilot -config file:///local/containerpilot.json /run.sh

[dvusboy]

This suggest you don't understand the difference between ENTRYPOINT and CMD in a Dockerfile. Think of the values of command and args combine to form the values to replace CMD specified in the Dockerfile. They have no effect on the ENTRYPOINT.

[rickardrosen]

Entrypoint is clearly mentioned here:
https://github.com/hashicorp/nomad/blob/e9961ef728a19d3ce8a5d45f965df869bfd4ff3f/client/driver/docker.go#L113

There seems to be some general confusion regarding the differences as well. Perhaps you can enlighten me regarding the actual differences?

I believe nomad should be able to utilize the --entrypoint flag for docker run if requested.

[dadgar]

@rickardrosen Here is a pretty good explanation of the difference: http://stackoverflow.com/questions/21553353/what-is-the-difference-between-cmd-and-entrypoint-in-a-dockerfile

Entrypoint is useful when treating the container as a binary and that is why you are seeing the command/args being appended.

Is your ask to have entry point be overridable?

[rickardrosen]

Ideally I want to be able to tell the container exactly what to do, when scheduling it.

Our current use case is as follows:
We use docker MACVLAN network driver, which enables us to assign each container an individual IP, but nomad will only register the service with the consul agent IP.
To work around this issue we push down containerpilot as an artifact and set "nomad unique host name" as an Env var.
This alllows us to interpolate the consul agent in the containerpilot config to the one running on the nomad host, and the container will register itself using its own IP.
Works great, as long as there is not an entrypoint...

As it stands now, I will have to build a custom version of a container if it contains an entrypoint.

In general I think that being able to override the entrypoint would be a nice feature, or even an expected one, when setting command and args through a job.

[dadgar]

@rickardrosen I will mark it as an enhancement and hopefully get it in soon! As for expected behavior, I don't the we will ever override the entry point because it is usually set when building the container so it behaves in a particular way!

Either way we will get this in and hopefully make your life a bit easer 😄

[rickardrosen]

Thanks Dadgar!
I completely understand the rationale behind the entrypoint.
The thing is that when scheduling something on my infrastructure I'm the one that want to decide the particulars and behavior of the container. :)

The reason I ended up here is that we're forced to use containerpilot.
Being able to override entrypoint would be great, but even better would be for nomad to be able to inspect and register the container to a service using the IP of the container.
Is this somewhere on the map?

[dadgar]

Yes we would like to solve that but it I do not have a date to give out on that one! Sorry!

[dvusboy]

@rickardrosen I'm still confused about the ask. If you want to invoke whatever you provide under "command" and "args" to the Nomad task, then you'd not want to set an ENTRYPOINT in the Dockerfile. Do you not have control over the Docker image?

[rickardrosen]

@dvusboy: exactly, it is not my image, and I'd prefer not keep a registry with modified containers, just to inject containerpilot.

[dvusboy]

In that case, it sounds like you're asking for the support of --entrypoint to the Docker driver. I don't believe treating "command" as an implicit override to the ENTRYPOINT of the image is intuitive.

[rickardrosen]

Well it depends on how you view it.
If I spec a "command" through my generic scheduler I would expect this command to actually run!
If you look at it from a docker perspective, sure I'm asking for entrypoint support.

Either way, the end result will be the same: the specified command will be run.

[ryanmickler]

ah this would be a great feature! #

[dvusboy]

@rickardrosen That part really depends on the implementation of the entrypoint to the Docker image. Some scripts allow pass-thru, some don't. You can't expect Nomad to control this outside of providing explicit support of --entrypoint.

[rickardrosen]

@dvusboy I have no such expectations =)

Support for --entrypoint would be perfect for our scenario as it will allow overriding whatever our devs and others want to run, prepending that with containerpilot when scheduling.

[schmichael]

Just wanted to note that Nomad 0.6 will advertise the Docker network driver IP automatically, so containerpilot shouldn't be necessary: https://www.nomadproject.io/docs/job-specification/service.html#address_mode

Leaving this issue open as its for a more generic feature than just customizing advertisement.

[dvusboy]

@schmichael Does that include the built-in bridge driver?

[schmichael]

@dvusboy bridge and host will advertise the host's IP by default. To advertise the bridge IP you have to set address_mode=driver on the service.

[dvusboy]

Thanks, @schmichael, the code is a bit dense to trace. So, if I'm using the Docker driver, and under task : service, I specify address_mode = "driver". Then Nomad will register the container's IP address to Consul service advertisement, correct? Or do you mean the IP address of the Linux bridge/virtual switch is published.

[schmichael]

It is unfortunately dense, and the docs could definitely use expanding. I don't think this little blurb helps either: https://www.nomadproject.io/docs/drivers/docker.html#advertising-container-ips Docs PRs always welcome, but I'll see if there's anything I can do.

if I'm using the Docker driver, and under task : service, I specify address_mode = "driver". Then Nomad will register the container's IP address to Consul service advertisement, correct?

Correct!

The default of auto just means:

• Use host's IP for host and bridge
• Use container's IP for (other/custom) network plugins

[bittrance]

One use case where one wants to override the ENTRYPOINT is when you want to add timeout binary as Nomad has no support for terminating batch jobs that intermittently take too long, see #1782 .

[ericcancil]

Do we have any movement on this?

[samart]

ak, I need to override entrypoint as well...

[dadgar]

Hey all we are looking into this for 0.8