Prometheus: consul service discvery: I expect use host address instead of ServiceAddress

[yteraoka]

What did you do?

I tried consul service discovery.

What did you expect to see?

Prometheus server that in outside of docker cuslter access exposed port on docker host IP address.

What did you see instead? Under which circumstances?

Consul service discovery use ServiceAddress that in docker internal network.
So, Prometheus server can't reach metrics endpoint.

Current implementation of consul service discovery use ServiceAddress prior to address.
https://github.com/prometheus/prometheus/blob/master/discovery/consul/consul.go#L278

Consul return to prometheus bellow response.

[
  {
    "Node": "docker1",
    "Address": "192.168.2.97", # Host IP
    "ServiceID": "docker1:testapp:8080",
    "ServiceName": "test",
    "ServiceTags": [],
    "ServiceAddress": "10.0.0.3",  # Container IP
    "ServicePort": 32814,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 9033587,
    "ModifyIndex": 9033587
  }
]

Prometheus server is running outside of docker cluster.
Server cant't reach ServiceAddress that in docker internal network.
So, I expect using host address instead of ServiceAddress.

Is there a way to solve this issue?

Environment

Docker 1.11 + swarm with consul.
Registrator store container information into consul.

• System information:

# uname -srm
Linux 3.10.0-327.36.2.el7.x86_64 x86_64

• Prometheus version:

# ./prometheus  -version
prometheus, version 1.4.1 (branch: master, revision: 2a89e8733f240d3cd57a6520b52c36ac4744ce12)
  build user:       root@e685d23d8809
  build date:       20161128-09:59:22
  go version:       go1.7.3

• Prometheus configuration file:

  - job_name: 'consul-test'
    metrics_path: '/api/prometheus'
    consul_sd_configs:
      - server: '127.0.0.1:8500'
        datacenter: 'dc1'
        services:
          - 'testapp'

• Logs:

strace output

[pid 22676] connect(33, {sa_family=AF_INET, sin_port=htons(32814), sin_addr=inet_addr("10.0.0.3")}, 16) = -1 EINPROGRESS (Operation now in progress)

[brian-brazil]

You can use relabelling to pull in the other address.

[yteraoka]

@brian-brazil thanks!!

This issue solved with relabel config below. :+1:

    relabel_configs:
      - source_labels: ['__meta_consul_address', '__meta_consul_service_port']
        separator: ':'
        regex: '(.*)'
        target_label: '__address__'
        replacement: '$1'

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.