Swarmkit: [Proposal] docker service create template support for {{ .Node.Hostname }}

[approxit]

Background

Templating support gives a massive use cases and flexibility. Having control over for eg. service container hostname is very useful for some sort of tools.

Let's take for eg. Sentry - error tracking tool.
We hook up integration to our app and everything that goes wrong will be seen from there in report-like view, with all details you need. One of field that you can filter is server_name which is actually a hostname of the running container which by default is contained id.

And here things get a little messy: We're have no direct clue about on which server task was running, either container have no clue about on what server is running. On swarm solutions this information is quite useful - take notes about global services.

What docker gives us now?

• Current template implementation have {{ .Node.Id }} available to use. It's a intermediate value, given by docker, which must be "decoded" by human to receive information about "on which server that service task was running". docker node ls or similar commands are required to check it out. (Cross your fingers if that server is still available in cluster and can be listed!)

The goal of this proposal

• To add{{ .Node.Hostname }} which resolve these issues and give more control over service awareness about on which server task is running.

The use cases

• Services that are bound to server instances are aware about them. "Your application event failed on AppSrv3 due to not enough free disc space"
• Task hostnames are managed by sysops, and machines hostnames are their natural identifier. "Hay, guys, we need take actions on AppSrv3. Fortunately we don't need to browse through these node ids manually!"

docker/docker#28486

[gianarb]

I think that's a good idea. Hostname is easy to customize and human readable. I can take care of this implementation is @approxit is not interested to work on it directly and we are happy to add this variable.

[approxit]

I'm happy to hear that, @gianarb. Totally go for it's implementation.

[gianarb]

It's not that easy because the service's hostname is not into the ctx. ctx is implement to deal with the the task itself. It's pretty clear because the function name is NewContextFromTask.

I followed the function back and I can not see a good place to add the Service in the scope. Maybe @aaronlehmann or @stevvooe can help up to figure out the right path.

[aaronlehmann]

I'm confused. Why would adding {{ .Node.Hostname }} involve the service?

[gianarb]

eg. service container hostname is very useful for some sort of tools.

Maybe I misunderstood. I started to work with the os.Hostname but the issue is speaking about service hostname. it means probably

swarmkit service create --hostname hello

But when I originally read this issue I was thinking about the os hostname :confused:

[stevvooe]

This issue is asking for populating templatable fields with the Node's hostname, not changing the hostname of the task (which is already possible).

You could always add new information to build the context. The only thing that would need to be done here is to ensure that the context for the template got the same value for Node.Hostname as it did for docker node ls. Just make sure you don't make loops possible or likely in the future.

[gianarb]

@approxit I am going to implement the node's hostname :+1:

[gr4per]

+1 will this be available in docker service create ?

[approxit]

We have a proposal, we have a PR, what are the next steps?

[aaronlehmann]

We have a proposal, we have a PR, what are the next steps?

Where is the PR?

[gianarb]

@approxit the PR is not ready :) that's why it's not there. If you have time feel free to update. i did some tests few weeks ago but it was not easy as expected

[mostolog]

any news?

[cjbearman]

I'd like to add a +1 to this feature request.

The ability to have a template value that equates to the node's hostname would be extremely useful for scenarios involving monitoring. One specific use case that I have encountered is that I am using Google cAdvisor to capture usage stats as a global service within my swarm. The data is then sent to Influx and visualized using Grafana as described here.

The "host" shown in Grafana is derived from the hostname of the cadvisor container. Right now the best I can do is to template that to the node ID. Instead of seeing friendly node hostnames, I see not so friendly node IDs for all my data.

I will note, for the benefit of others, that a potential workaround is to bind mount /etc/hostname into your container at an alternative path (say /etc/nodehostname) and then read the node's hostname from that file. Unfortunately for me that does not work since cAdvisor will only utilize the node's actual hostname as determined through the kernel.

So for me, a definite 1+ on this request.

[sadok-f]

+1

[sugun999]

+1

[m8o3O8m]

👍

[gianarb]

THis is done https://github.com/docker/swarmkit/pull/2266

[artisticcheese]

Which version of docker will this appear on?

[realcbb]

Which version of docker will this appear on?

[stevvooe]

@thaJeztah ^

[thaJeztah]

I see it was vendored in through https://github.com/moby/moby/pull/34061, which is included in docker 17.07; https://github.com/docker/docker-ce/commit/57481ff30bbddefc30c2c5613f373b5faaed7eef

I'll have to check if the documentation is updated accordingly, and if there's changes needed in the CLI (or if they were made already)

[thaJeztah]

Does not seem to be working on 17.07-rc1;

docker service create \
  --name test-1951 \
  --hostname="{{.Node.Hostname}}-{{.Service.Name}}"\
   busybox top


docker inspect --format '{{.Config.Hostname}}' 70fff4069523
-test-1951

TaskTemplate looks correct;

        "Spec": {
            "ContainerSpec": {
                "Image": "busybox:latest@sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4",
                "Args": [
                    "top"
                ],
                "Hostname": "{{.Node.Hostname}}-{{.Service.Name}}",
                "DNSConfig": {}
            },

Node inspect gives me;

[
    {
        "ID": "drfinwj2um6lv0pbprh0gzzfm",
        "Version": {
            "Index": 88687
        },
        "CreatedAt": "2017-01-16T10:05:50.696818052Z",
        "UpdatedAt": "2017-08-08T00:19:22.111542951Z",
        "Spec": {
            "Labels": {
                "node.labels.az": "foo"
            },
            "Role": "manager",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "moby",
            "Platform": {
                "Architecture": "x86_64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 4000000000,
                "MemoryBytes": 2095882240
            },

Wondering if the template should use Node.Description.Hostname instead, or if an update is needed to the Node API type

edit: nope, that's not it:

docker service create \
  --name test-1951 \
  --hostname="{{.Node.Description.Hostname}}-{{.Service.Name}}"\
  busybox top

Error response from daemon: rpc error: code = InvalidArgument desc = expanding hostname failed: template: expansion:1:7: executing "expansion" at <.Node.Description.Ho...>: can't evaluate field Description in type struct { ID string; Hostname string; Platform template.Platform }

[thaJeztah]

Interesting; so the only place the new functionality is called is in; https://github.com/moby/moby/blob/1009e6a40b295187e038b67e184e9c0384d95538/daemon/cluster/executor/container/container.go#L80-L88

However information about the Node where that's executed is nowhere around there (unless I miss something). api.Task only has NodeID, no other information about the node

// Task specifies the parameters for implementing a Spec. A task is effectively
// immutable and idempotent. Once it is dispatched to a node, it will not be
// dispatched to another node.
type Task struct {
    ID   string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
    Meta Meta   `protobuf:"bytes,2,opt,name=meta" json:"meta"`
    // Spec defines the desired state of the task as specified by the user.
    // The system will honor this and will *never* modify it.
    Spec TaskSpec `protobuf:"bytes,3,opt,name=spec" json:"spec"`
    // SpecVersion is copied from Service, to identify which version of the
    // spec this task has. Note that this is not directly comparable to the
    // service's Version.
    SpecVersion *Version `protobuf:"bytes,14,opt,name=spec_version,json=specVersion" json:"spec_version,omitempty"`
    // ServiceID indicates the service under which this task is orchestrated. This
    // should almost always be set.
    ServiceID string `protobuf:"bytes,4,opt,name=service_id,json=serviceId,proto3" json:"service_id,omitempty"`
    // Slot is the service slot number for a task.
    // For example, if a replicated service has replicas = 2, there will be a
    // task with slot = 1, and another with slot = 2.
    Slot uint64 `protobuf:"varint,5,opt,name=slot,proto3" json:"slot,omitempty"`
    // NodeID indicates the node to which the task is assigned. If this field
    // is empty or not set, the task is unassigned.
    NodeID string `protobuf:"bytes,6,opt,name=node_id,json=nodeId,proto3" json:"node_id,omitempty"`
    // Annotations defines the names and labels for the runtime, as set by
    // the cluster manager.
    //
    // As backup, if this field has an empty name, the runtime will
    // allocate a unique name for the actual container.
    //
    // NOTE(stevvooe): The preserves the ability for us to making naming
    // decisions for tasks in orchestrator, albeit, this is left empty for now.
    Annotations Annotations `protobuf:"bytes,7,opt,name=annotations" json:"annotations"`
    // ServiceAnnotations is a direct copy of the service name and labels when
    // this task is created.
    //
    // Labels set here will *not* be propagated to the runtime target, such as a
    // container. Use labels on the runtime target for that purpose.
    ServiceAnnotations Annotations `protobuf:"bytes,8,opt,name=service_annotations,json=serviceAnnotations" json:"service_annotations"`
    Status             TaskStatus  `protobuf:"bytes,9,opt,name=status" json:"status"`
    // DesiredState is the target state for the task. It is set to
    // TaskStateRunning when a task is first created, and changed to
    // TaskStateShutdown if the manager wants to terminate the task. This field
    // is only written by the manager.
    DesiredState TaskState `protobuf:"varint,10,opt,name=desired_state,json=desiredState,proto3,enum=docker.swarmkit.v1.TaskState" json:"desired_state,omitempty"`
    // List of network attachments by the task.
    Networks []*NetworkAttachment `protobuf:"bytes,11,rep,name=networks" json:"networks,omitempty"`
    // A copy of runtime state of service endpoint from Service
    // object to be distributed to agents as part of the task.
    Endpoint *Endpoint `protobuf:"bytes,12,opt,name=endpoint" json:"endpoint,omitempty"`
    // LogDriver specifies the selected log driver to use for the task. Agent
    // processes should always favor the value in this field.
    //
    // If present in the TaskSpec, this will be a copy of that value. The
    // orchestrator may choose to insert a value here, which should be honored,
    // such a cluster default or policy-based value.
    //
    // If not present, the daemon's default will be used.
    LogDriver                *Driver            `protobuf:"bytes,13,opt,name=log_driver,json=logDriver" json:"log_driver,omitempty"`
    AssignedGenericResources []*GenericResource `protobuf:"bytes,15,rep,name=assigned_generic_resources,json=assignedGenericResources" json:"assigned_generic_resources,omitempty"`
}

[stevvooe]

@thaJeztah Here is the context for templates. For this change to take effect, the hostname had to be correctly injected, but it looks like the vendoring just passed in nil: https://github.com/moby/moby/pull/34061/files#diff-9651468cafd3eee09570f3733efd7dc4L80.

[aaronlehmann]

That's correct, the injection has not been added to the moby executor yet.

[mion00]

I think that this issue is resolved, given the latest pull request that have been merged in the Moby repo.

[stevvooe]

@mion00 Confirmed. Thanks!

[jmarcos-cano]

What exact version contains this ?

[thaJeztah]

Tested on master;

$ hostname
thisismyhost

$ docker service create \
  --name test-1951 \
  --hostname="{{.Node.Hostname}}-{{.Service.Name}}"\
   busybox top

$ docker inspect --format '{{.Config.Hostname}}' test-1951.1.kgzsdyd6wogug8yz8qyvwmmg8
thisismyhost-test-1951

@jmarcos-cano This is currently on master, and will be included in docker 17.10

Closing this one because it's resolved, thanks!

[approxit]

Thanks a lot for all the effort to make this feature happen!