Marathon: entrypoint overwritten for docker containers

[a-nldisr]

Hello,

Recently we updated from marathon 0.10.0 to 0.11.0
Today i tried to restart a Marathon / Docker config we had running earlier. The job however respawned every time as if it ran successfully. The stdout and stderr were empty and the docker containers ran for 1 second.

This is what i saw in docker ps -a

docker ps -a
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS                           PORTS               NAMES
283a72643a36        buildserver.surfsara.nl:5000/hdfs-nn:latest   "/bin/sh -c ' '"         4 minutes ago       Exited (0) 4 minutes ago                             mesos-20150630-143338-1545979914-5050-6950-S83.7b6ca5d0-411a-4e02-aed8-20191378757a
48cbac75ad5d        buildserver.surfsara.nl:5000/hdfs-nn:latest   "/bin/sh -c ' '"         4 minutes ago       Exited (0) 4 minutes ago                             mesos-20150630-143338-1545979914-5050-6950-S83.c9be6392-552f-4750-96ef-8c6acf3a3e54

I was missing something, the part from the entrypoint i specified in the container. When i start the container from the command line i see this:

7335f29e4b78        buildserver.surfsara.nl:5000/hdfs-nn:latest   "/bin/sh -c '/opt/bin"   14 minutes ago      Exited (137) 10 minutes ago                          elated_brattain

How the entrypoint is specified in the Dockerfile:
ENTRYPOINT /opt/bin/pre-start-hadoop.sh && /usr/bin/supervisord

I suspect there is something going wrong with sanitizing the input. I can start the container when i specify the command above used in the entrypoint.

[gkleiman]

Can you please add the json app definition that you get from GET /v2/apps/?

[VFT]

with v0.10.0,marathon will set default cmd as that initial cmd from docker image,but v0.11.0 it will set cmd as "/bin/sh -c ' ' " if you set the "cmmond" textput to empty

so,you need to set cmd manually

[DavidWittman]

I'm having this issue as well. It looks like when adding tasks via the UI, command is set to a single space, which is causing the entrypoint to be overridden as described here. Here's the output from GET /v2/apps:

{
    "apps": [
        {
            "acceptedResourceRoles": null,
            "args": null,
            "backoffFactor": 1.15,
            "backoffSeconds": 1,
            "cmd": " ",
            "constraints": [],
            "container": {
                "docker": {
                    "forcePullImage": false,
                    "image": "objectrocket/kibana:latest",
                    "network": "BRIDGE",
                    "parameters": [],
                    "portMappings": [
                        {
                            "containerPort": 5601,
                            "hostPort": 0,
                            "protocol": "tcp",
                            "servicePort": 10000
                        }
                    ],
                    "privileged": false
                },
                "type": "DOCKER",
                "volumes": []
            },
            "cpus": 1.0,
            "dependencies": [],
            "deployments": [],
            "disk": 256.0,
            "env": {
                "ELASTICSEARCH_URL": "<redacted>"
            },
            "executor": "",
            "healthChecks": [],
            "id": "/kibana",
            "instances": 1,
            "labels": {},
            "maxLaunchDelaySeconds": 3600,
            "mem": 512.0,
            "ports": [
                10000
            ],
            "requirePorts": false,
            "storeUrls": [],
            "tasksHealthy": 0,
            "tasksRunning": 0,
            "tasksStaged": 0,
            "tasksUnhealthy": 0,
            "upgradeStrategy": {
                "maximumOverCapacity": 1.0,
                "minimumHealthCapacity": 1.0
            },
            "uris": [],
            "user": null,
            "version": "2015-10-14T23:27:24.141Z",
            "versionInfo": {
                "lastConfigChangeAt": "2015-10-14T23:27:24.141Z",
                "lastScalingAt": "2015-10-14T23:27:24.141Z"
            }
        }
    ]
}

[jstabenow]

Here an example to produce this problem:

curl marathon:8080/v2/apps/nginx

{
    "app": {
        "id": "/nginx",
        "cmd": " ",
        "args": null,
        ...

[jstabenow]

I'm not a programmer, but that looks to me suspiciously ;-)

Marathon-UI > AppsFromModelPostProcess.js:

// Remove this hack, if there is a solution available.
// https://github.com/mesosphere/marathon/issues/2147
if (Util.isEmptyString(app.cmd)) {
   app.cmd = " ";
}

[jstabenow]

Same issue?: #2147

[DavidWittman]

Yeah, this looks like a dupe.

edit: Or rather, an unintended side effect.

[abridgett]

2147 looks done now, it looks as if we need to have the option to send this as "args" rather than "cmd" to the API (https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps). Currently AFAICT this breaks all images with an entrypoint (especially those without /bin/sh). Actually it's a bit more involved as args needs to be an array so you have to deal with splitting on whitespace, quoting etc.

[aquamatthias]

@pierlo-upitup can this one be closed?

[pierluigi]

If we intend this issue as a dupe of #2147 then, yes – that was fixed in 0.13, where the cmd field is not set to (whitespace) anymore. https://github.com/mesosphere/marathon-ui/pull/460

[justinclayton]

I would say this should not be closed. Forcing entrypoint to /bin/sh -c for everyone is bad for a great number of reasons:

1. it overrides the intended behavior of the container image creator
2. it's overridden to something that does not exist in all containers
3. it's overridden to something that does not allow for SIGTERM to be trapped by children

The most frustrating (and amusing) bit I discovered is that you can't opt-out of it. Try passing "parameters":[{"key":"entrypoint","value":"/my-original-entrypoint"}] and enjoy the lulz as you find that what it actually passed on the command line for docker run was --entrypoint=/my-original-entrypoint --entrypoint /bin/sh. Then it tosses -c to the front of the command section, which can also lead to very unexpected behavior.

Bottom line, Marathon shouldn't have a horse in this race IMHO; it should simply honor the container ecosystem's options for configuration and not force the user to jump through hoops to work around this.

[polym]

Met same problem.

root     21972  1.5  0.2 133684 15720 ?        Sl   10:29   0:00 docker -H 
unix:///var/run/docker.sock run --cpu-shares 102 --memory 33554432 
-e MARATHON_APP_VERSION=2016-02-26T15:02:10.751Z -e HOST=10.0.1.193 -e MARATHON_APP_RESOURCE_CPUS=0.1
-e MARATHON_APP_DOCKER_IMAGE=gitlab/gitlab-runner -e MESOS_TASK_ID=gitlab-ci-runner.ab1d0505-dc9d-11e5-9668-6eb5f3e16a90 
-e MARATHON_APP_RESOURCE_MEM=16.0 -e MARATHON_APP_RESOURCE_DISK=0.0 
-e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/gitlab-ci-runner 
-e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0.342b695d-7078-4c8a-9bbb-2098cd2281d9 
-v /var/run/docker.sock:/var/run/docker.sock:rw -v /etc/gitlab-runner:/srv/gitlab-runner/config:ro 
-v /usr/bin/docker:/usr/bin/docker.static:ro -v /tmp/mesos/slaves/80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0/frameworks/b0c582c1-1530-451f-95ed-537beaa5a251-0000/executors/gitlab-ci-runner.ab1d0505-dc9d-11e5-9668-6eb5f3e16a90/runs/342b695d-7078-4c8a-9bbb-2098cd2281d9:/mnt/mesos/sandbox --net bridge 

--entrypoint=/docker-entrypoint.sh --entrypoint /bin/sh

--name mesos-80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0.342b695d-7078-4c8a-9bbb-2098cd2281d9 gitlab/gitlab-runner -c run --user=root --working-directory=/home/gitlab-runner

[justinclayton]

@polym Hey, gitlab-runner is exactly what I'm trying to get working too! Nice.

So, I'm going to eat a little crow here instead of just deleting my earlier post. For others who are banging their heads against this same wall: it turns out this isn't an outstanding Marathon issue at all. At worst it's obscure and in need of clearer documentation.

If you want to preserve a container's custom entrypoint, you need to remove cmd from the top level of the app definition and add "args":[] to the "container" object.

Using cmd sets CommandInfo.shell to true when Marathon calls the Mesos containerizer, and this setting is what overrides the entrypoint to /bin/sh and prepends -c to the Docker container's CMD. Using args sets CommandInfo.shell to false, which results in simply passing args into CMD as is. The latter is probably what you want most of the time.

Pro tip: if you want to override the entrypoint yourself, you are now able to do that by passing parameters, like so:

{
  "id": "/gitlab-runner",
  "fetch": [
    { "uri": "https://path/to/custom/script/runner.sh", "executable": true }
  ],
  "container": {
    "type": "DOCKER",
    "args": [],
    "docker": {
      "image": "gitlab/gitlab-runner",
      "network": "BRIDGE",
      "parameters": [
        { "key": "entrypoint", "value": "/mnt/mesos/sandbox/runner.sh" }
      ]
    }
  }
}

I'm still having trouble with my SIGTERMs getting trapped, but at least I can now make apples to apples comparisons with my local setup.

[xiajidexue]

@justinclayton, i met the same problem using Jenkins on Mesos plugin, where should i set CommandInfo.shell to false? I did not know how to set this

[ahmet2mir]

@justinclayton works well, but you have to use REST API, the UI (at least 1.3.6) will remove args and force cmd to " " in JSON mode

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-3634. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.