Nomad: Connect stanza interpolation limitations

[schmichael]

Nomad version

v0.10.4

Issue

Interpolation is still limited within group.service.connect stanzas because the majority of interpolation is done at the task level by the TaskEnv struct.

Hopefully this can be fixed without HCL2 (#7220).

Reproduction steps

Run jobspec from gist: https://gist.github.com/schmichael/dfdb9984c549ff4f5c65082f24fdf874

Observe that in nomad alloc status output the meta var is interpolated in the service name, but not in anything Connect related:

Name                   = countdash.api[0]
Job ID                 = countdash
Client Status          = failed
Client Description     = Failed tasks
...

Allocation Addresses (mode = "bridge")
Label                                         Dynamic  Address
connect-proxy-${NOMAD_META_api_service_name}  yes      192.168.86.127:23331 -> 23331

Task "connect-proxy-${NOMAD_META_api_service_name}" is "dead"
...

Recent Events:
Time                       Type           Description
2020-02-21T09:43:33-08:00  Setup Failure  failed to setup alloc: pre-run hook "group_services" failed: invalid Consul Connect configuration for service "count-api": No Connect port defined for service "count-api"
2020-02-21T09:43:31-08:00  Received       Task received by client

Task "web" is "dead"
...

Recent Events:
Time                       Type           Description
2020-02-21T09:43:33-08:00  Setup Failure  failed to setup alloc: pre-run hook "group_services" failed: invalid Consul Connect configuration for service "count-api": No Connect port defined for service "count-api"
2020-02-21T09:43:31-08:00  Received       Task received by client

[jorgemarey]

Hi! Maybe this doens't fit in this issue exactly, but I found out a similar problem when using interpolation in the service name at the group level.

failed to setup alloc: pre-run hook "group_services" failed: invalid Consul Connect configuration for service "jobname-service": No Connect port defined for service "jobname-service"

    group "my-group" {
      network {
        mode ="bridge"
        port "http" {
           to = 8443
         }
      }
      service {
        name      = "${NOMAD_JOB_NAME}-service"
        port         = "8443"
        connect {
          sidecar_service {}
        }
        ...
    }
...
}

[trilom]

Pretty sure this problem fits here, but it's a little annoying to pass configuration to a service stanza without explicitly writing out the whole task stanza, any solutions to this?

Below is configuration snippet for service, and the output on envoy connect container.

    service {
      name = "api"
      tags = ["global","app"]
      port = "9091"

      connect {
        sidecar_service {
        proxy {
            local_service_address = "${attr.unique.network.ip-address}"
          }
        }
      }
    }

[2020-07-21 17:54:15.618][1][warning][config] [source/common/config/grpc_mux_subscription_impl.cc:72] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) local_app: malformed IP address: ${attr.unique.network.ip-address}. Consider setting resolver_name or setting cluster type to 'STRICT_DNS' or 'LOGICAL_DNS'

[jharley]

Maybe this doens't fit in this issue exactly, but I found out a similar problem when using interpolation in the service name at the group level.

Can confirm it does. I just chased my own tail a bit with the following "service" stanza:

    service {
      name = "${NOMAD_GROUP_NAME}"
      port = "5000"

      connect {
        sidecar_service {}
      }
    }

And got: pre-run hook "group_services" failed: invalid Consul Connect configuration for service "myservice": No Connect port defined for service "myservice"

Replacing ${NOMAD_GROUP_NAME} with myservice resolved the issue

[tgross]

Another case of this looks to be that the connect.sidecar.proxy.config doesn't get interpolated. See @idrennanvmware's comment here: https://github.com/hashicorp/nomad/issues/9161#issuecomment-716534426

[idrennanvmware]

Looks like exactly the same as @trilom issue. Thanks all!

[rickardrosen]

I stumbled upon this when trying to expose the envoy prometheus metrics - interpolation does not work for proxy task tags. (Is there a way to do this without exposing a dynamic port on the host? I can't really find any docs on this.)

If I set prometheus tags on the proxy task I can scrape metrics, but when using the host interface I have to (?) interpolate the host ip and dynamic port. Here's a non-working excerpt:

                        "Connect": {
                            "Native": false,
                            "SidecarService": {
                                "Port": "",
                                "Proxy": {
                                    "Config": null,
                                    "ExposeConfig": {
                                        "Path": [
                                            {
                                                "Path": "/health",
                                                "Protocol": "http",
                                                "LocalPathPort": 80,
                                                "ListenerPort": "healthcheck"
                                            },
                                            {
                                                "Path": "/metrics",
                                                "Protocol": "http",
                                                "LocalPathPort": 80,
                                                "ListenerPort": "metrics"
                                            },
                                            {
                                                "Path": "/stats/prometheus",
                                                "Protocol": "http",
                                                "LocalPathPort": 19001,
                                                "ListenerPort": "envoy_metrics"
                                            }
                                        ]
                                    },
                                    "Upstreams": [
                                        {
                                            ...
                                        }
                                    ]
                                },
                                "Tags": [
                                    "prometheus_metrics_address=${attr.unique.network.ip-address}:${NOMAD_PORT_envoy_metrics}",
                                    "prometheus_metrics_path=/stats/prometheus",
                                    "prometheus_metrics=true"
                                ]

[jorgemarey]

Hi @rickardrosen. This is not directly related with this issue, but I'm trying to get a PR merged in prometheus to be able to get relabel info for connect exposeConfig in here. I think that would solve this issue for you as prometheus would be able to use the dynamic port.