Ingress-nginx: prometheus metrics names & grafana dashboard availability

[arjanschaaf]

We have been running a setup with this nginx ingress controller in combination with a prometheus deployment. When we started out, we rolled out an additional container running the https://github.com/hnlq715/nginx-vts-exporter/ container which export the data from :18080/nginx_status/format/json into prometheus. This works good enough including a nice basic dashboard containing the data: https://grafana.com/dashboards/2949

But since the nginx ingress controller exposes the prometheus data on :10254/metrics itself as documented here: https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/custom-vts-metrics-prometheus/README.md#prometheus-metrics-output I thought: why not get rid of the obsolete hnlq715/nginx-vts-exporter setup and let Prometheus scrape the nginx-ingress-controller directly on :10254/metrics.

But since the metrics names created on :10254/metrics do not match the metrics from hnlq715/nginx-vts-exporter the grafana dashboard isn't going to work.

So I wondered if anybody created an grafana dashboard that can be used with the data stored in Prometheus by the nginx-ingress-controller :10254/metrics endpoint?

[aledbf]

@arjanschaaf I am building a Grafana dashboard. Before the next release, I will add the json file and a mention in the docs.

[arjanschaaf]

Excellent: looking forward to it! 😄

[azman0101]

Hi,

I've fix Nginx VTS Stats which may certainly be improved:

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS-K8S",
      "label": "prometheus-k8s",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "4.4.3"
    },
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": ""
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "1.0.0"
    }
  ],
  "annotations": {
    "list": []
  },
  "description": "Show stats from the nginx-ingress.",
  "editable": true,
  "gnetId": 1623,
  "graphTooltip": 0,
  "hideControls": false,
  "id": null,
  "links": [],
  "refresh": false,
  "rows": [
    {
      "collapse": false,
      "height": 248,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 1,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 6,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_connections_total[5m])) by (type)",
              "format": "time_series",
              "intervalFactor": 2,
              "legendFormat": "{{ type }}",
              "metric": "nginx_server_connections",
              "refId": "A",
              "step": 4
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Server Connections",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 2,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 6,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_bytes_total{server_zone=~\"^$Host$\"}[5m])) by (direction)",
              "format": "time_series",
              "intervalFactor": 2,
              "legendFormat": "{{ direction }}",
              "metric": "nginx_server_bytes",
              "refId": "A",
              "step": 4
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Server Bytes",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "bytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": "0",
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 298,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 8,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 6,
          "stack": true,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_responses_total{server_zone=~\"$Host\"}[5m])) by (status_code)",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
              "legendFormat": "{{ status_code }}",
              "metric": "nginx_responses_total",
              "refId": "A",
              "step": 4
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Server Responses",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 4,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 2,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_cache_total{server_zone=~\"^$Host$\"}[5m])) by (type)",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
              "legendFormat": "{{ type }}",
              "metric": "nginx_server_cache",
              "refId": "A",
              "step": 10
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Server Cache",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": "0",
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 3,
          "legend": {
            "alignAsTable": true,
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": false,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 4,
          "stack": true,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_requests_total{server_zone=~\"$Host\"}[5m])) by (server_zone)",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
              "legendFormat": "{{ server_zone }}",
              "metric": "nginx_server_requests",
              "refId": "A",
              "step": 4
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Server Requests",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 250,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 6,
          "legend": {
            "alignAsTable": true,
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": false,
            "show": false,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 6,
          "stack": true,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_upstream_requests_total{upstream=~\"^$Upstream$\"}[5m])) by (upstream)",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
              "legendFormat": "{{ upstream }}",
              "metric": "nginx_upstream_requests",
              "refId": "A",
              "step": 4
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Upstream Requests",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 5,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 6,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(irate(nginx_upstream_bytes_total{upstream=~\"^$Upstream$\"}[5m])) by (direction)",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
              "legendFormat": "{{ direction }}",
              "metric": "nginx_upstream_bytes",
              "refId": "A",
              "step": 4
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Upstream Bytes",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 250,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "${DS_PROMETHEUS-K8S}",
          "fill": 1,
          "id": 7,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 12,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "sum(nginx_upstream_responses_total{upstream=~\"^$Upstream$\"}) by (server)",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
              "legendFormat": "{{ server }}",
              "metric": "nginx_upstream_response",
              "refId": "A",
              "step": 2
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Upstream Backend Response",
          "tooltip": {
            "msResolution": false,
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    }
  ],
  "schemaVersion": 14,
  "style": "dark",
  "tags": [
    "prometheus",
    "nginx",
    "OF"
  ],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS-K8S}",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "Host",
        "options": [],
        "query": "label_values(nginx_bytes_total, server_zone)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": ".*",
        "current": {},
        "datasource": "${DS_PROMETHEUS-K8S}",
        "hide": 0,
        "includeAll": true,
        "label": null,
        "multi": false,
        "name": "Upstream",
        "options": [],
        "query": "label_values(nginx_upstream_bytes_total, upstream)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-30m",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "Nginx VTS Stats",
  "version": 6
}

[rajatjindal]

looking forward for it.

[gianrubio]

@aledbf are you working on this issue? Otherwise I have time to build dashboards and prometheus alerts for nginx ingress

[aledbf]

Otherwise I have time to build dashboards and prometheus alerts for nginx ingress

That would be awesome if you have the time

[toolboc]

Here is a dump of the output from the :10254/metrics endpoint. This shows which metrics are being published for Prometheus with a description of each metric in the associated 'HELP' section. Most of these are 'go process' and 'go memory' stats. There are only a few metrics that are specific to ingress-nginx operations:
ingress_controller_config_last_reload_successfull, ingress_controller_config_last_reload_successfull_timestamp_seconds
ingress_controller_success.

I was hoping there would be more metrics specific to nginx-ingress operations. In any event, I would be happy to work on a grafana dashboard for the most important metrics. Please feel free to share any ideas of what you would like to see given the metrics that are available below.

# HELP cpu_seconds_total Cpu usage in seconds
# TYPE cpu_seconds_total counter
cpu_seconds_total 1659.0999999999997
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000274582
go_gc_duration_seconds{quantile="0.25"} 0.000437446
go_gc_duration_seconds{quantile="0.5"} 0.000595938
go_gc_duration_seconds{quantile="0.75"} 0.000793277
go_gc_duration_seconds{quantile="1"} 0.025237446
go_gc_duration_seconds_sum 1.117934362
go_gc_duration_seconds_count 1097
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 85
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.691344e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.80325696e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 787049
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.2028341e+07
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.0199424e+07
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.691344e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.83328e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.81632e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 27488
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 630784
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.06496e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5279277822351248e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 13316
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.2055829e+07
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 46092
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65536
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.09296e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.372691e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.933312e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.933312e+06
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 3.5023996e+07
# HELP ingress_controller_config_last_reload_successfull Whether the last configuration reload attemp was successfull.\n                Prometheus alert example:\n             alert: IngressControllerFailedReload \n         expr: ingress_controller_config_last_reload_successfull == 0\n            for: 10m
# TYPE ingress_controller_config_last_reload_successfull gauge
ingress_controller_config_last_reload_successfull 1
# HELP ingress_controller_config_last_reload_successfull_timestamp_seconds Timestamp of the last successfull configuration reload.
# TYPE ingress_controller_config_last_reload_successfull_timestamp_seconds gauge
ingress_controller_config_last_reload_successfull_timestamp_seconds 1.52792751e+09
# HELP ingress_controller_success DEPRECATED: use ingress_controller_config_last_reload_successfull_timestamp_seconds or ingress_controller_config_last_reload_successfull instead.\n                    Cumulative number of Ingress controller reload operations
# TYPE ingress_controller_success counter
ingress_controller_success{count="reloads"} 91
# HELP num_procs number of processes
# TYPE num_procs gauge
num_procs 7
# HELP oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE oldest_start_time_seconds gauge
oldest_start_time_seconds 1.527800405e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1670.85
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 26
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.9581312e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.527800406e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 8.33855488e+08
# HELP read_bytes_total number of bytes read
# TYPE read_bytes_total counter
read_bytes_total 3.0748672e+07
# HELP resident_memory_bytes number of bytes of memory in use
# TYPE resident_memory_bytes gauge
resident_memory_bytes 7.602176e+07
# HELP virtual_memory_bytes number of bytes of memory in use
# TYPE virtual_memory_bytes gauge
virtual_memory_bytes 9.71493376e+08
# HELP write_bytes_total number of bytes written
# TYPE write_bytes_total counter
write_bytes_total 2.236416e+06
etricsginx-ingress-controller-7d47954db-45g6g:/etc/nginx# curl localhost:10254/me
# HELP cpu_seconds_total Cpu usage in seconds
# TYPE cpu_seconds_total counter
cpu_seconds_total 1659.1599999999999
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000274582
go_gc_duration_seconds{quantile="0.25"} 0.000437446
go_gc_duration_seconds{quantile="0.5"} 0.000595938
go_gc_duration_seconds{quantile="0.75"} 0.000793277
go_gc_duration_seconds{quantile="1"} 0.025237446
go_gc_duration_seconds_sum 1.117934362
go_gc_duration_seconds_count 1097
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 83
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.870976e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.80505328e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 787049
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.2028919e+07
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.0199424e+07
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.870976e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.685824e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.963776e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 29009
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 573440
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.06496e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5279277822351248e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 13370
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.2057928e+07
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 46092
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65536
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.09296e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.372691e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.933312e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.933312e+06
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 3.5023996e+07
# HELP ingress_controller_config_last_reload_successfull Whether the last configuration reload attemp was successfull.\n                Prometheus alert example:\n             alert: IngressControllerFailedReload \n         expr: ingress_controller_config_last_reload_successfull == 0\n            for: 10m
# TYPE ingress_controller_config_last_reload_successfull gauge
ingress_controller_config_last_reload_successfull 1
# HELP ingress_controller_config_last_reload_successfull_timestamp_seconds Timestamp of the last successfull configuration reload.
# TYPE ingress_controller_config_last_reload_successfull_timestamp_seconds gauge
ingress_controller_config_last_reload_successfull_timestamp_seconds 1.52792751e+09
# HELP ingress_controller_success DEPRECATED: use ingress_controller_config_last_reload_successfull_timestamp_seconds or ingress_controller_config_last_reload_successfull instead.\n                    Cumulative number of Ingress controller reload operations
# TYPE ingress_controller_success counter
ingress_controller_success{count="reloads"} 91
# HELP num_procs number of processes
# TYPE num_procs gauge
num_procs 7
# HELP oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE oldest_start_time_seconds gauge
oldest_start_time_seconds 1.527800405e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1670.91
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 26
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.9581312e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.527800406e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 8.33855488e+08
# HELP read_bytes_total number of bytes read
# TYPE read_bytes_total counter
read_bytes_total 3.0748672e+07
# HELP resident_memory_bytes number of bytes of memory in use
# TYPE resident_memory_bytes gauge
resident_memory_bytes 7.5976704e+07
# HELP virtual_memory_bytes number of bytes of memory in use
# TYPE virtual_memory_bytes gauge
virtual_memory_bytes 9.71497472e+08
# HELP write_bytes_total number of bytes written
# TYPE write_bytes_total counter

[gianrubio]

Did you enable vts metrics?

On Sat, 2 Jun 2018 at 16:42, Paul DeCarlo [e-mail] wrote:

Here is a dump of the output from the :10254/metrics endpoint. This shows
which metrics are being published for Prometheus with a description for
each method in the 'HELP' section. Most of these are 'go process' and 'go
memory stat's. There are only three that are specific to ingress-nginx
operations ingress_controller_config_last_reload_successfull,
ingress_controller_config_last_reload_successfull_timestamp_seconds, and
ingress_controller_success. I was hoping there would be a bit more
specific to the nging-ingress operations, perhaps this is due to a
misconfiguration? In any event, I would be happy to work on a grafana
dashboard for the most important metrics. Please feel free to share any
ideas you would like to see.

HELP cpu_seconds_total Cpu usage in seconds

TYPE cpu_seconds_total counter

cpu_seconds_total 1659.0999999999997

HELP go_gc_duration_seconds A summary of the GC invocation durations.

TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 0.000274582
go_gc_duration_seconds{quantile="0.25"} 0.000437446
go_gc_duration_seconds{quantile="0.5"} 0.000595938
go_gc_duration_seconds{quantile="0.75"} 0.000793277
go_gc_duration_seconds{quantile="1"} 0.025237446
go_gc_duration_seconds_sum 1.117934362
go_gc_duration_seconds_count 1097

HELP go_goroutines Number of goroutines that currently exist.

TYPE go_goroutines gauge

go_goroutines 85

HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.

TYPE go_memstats_alloc_bytes gauge

go_memstats_alloc_bytes 3.691344e+06

HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.

TYPE go_memstats_alloc_bytes_total counter

go_memstats_alloc_bytes_total 9.80325696e+08

HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.

TYPE go_memstats_buck_hash_sys_bytes gauge

go_memstats_buck_hash_sys_bytes 787049

HELP go_memstats_frees_total Total number of frees.

TYPE go_memstats_frees_total counter

go_memstats_frees_total 1.2028341e+07

HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.

TYPE go_memstats_gc_sys_bytes gauge

go_memstats_gc_sys_bytes 2.0199424e+07

HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.

TYPE go_memstats_heap_alloc_bytes gauge

go_memstats_heap_alloc_bytes 3.691344e+06

HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.

TYPE go_memstats_heap_idle_bytes gauge

go_memstats_heap_idle_bytes 4.83328e+06

HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.

TYPE go_memstats_heap_inuse_bytes gauge

go_memstats_heap_inuse_bytes 5.81632e+06

HELP go_memstats_heap_objects Number of allocated objects.

TYPE go_memstats_heap_objects gauge

go_memstats_heap_objects 27488

HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.

TYPE go_memstats_heap_released_bytes_total counter

go_memstats_heap_released_bytes_total 630784

HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.

TYPE go_memstats_heap_sys_bytes gauge

go_memstats_heap_sys_bytes 1.06496e+07

HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.

TYPE go_memstats_last_gc_time_seconds gauge

go_memstats_last_gc_time_seconds 1.5279277822351248e+09

HELP go_memstats_lookups_total Total number of pointer lookups.

TYPE go_memstats_lookups_total counter

go_memstats_lookups_total 13316

HELP go_memstats_mallocs_total Total number of mallocs.

TYPE go_memstats_mallocs_total counter

go_memstats_mallocs_total 1.2055829e+07

HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.

TYPE go_memstats_mcache_inuse_bytes gauge

go_memstats_mcache_inuse_bytes 3472

HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.

TYPE go_memstats_mcache_sys_bytes gauge

go_memstats_mcache_sys_bytes 16384

HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.

TYPE go_memstats_mspan_inuse_bytes gauge

go_memstats_mspan_inuse_bytes 46092

HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.

TYPE go_memstats_mspan_sys_bytes gauge

go_memstats_mspan_sys_bytes 65536

HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.

TYPE go_memstats_next_gc_bytes gauge

go_memstats_next_gc_bytes 7.09296e+06

HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.

TYPE go_memstats_other_sys_bytes gauge

go_memstats_other_sys_bytes 1.372691e+06

HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.

TYPE go_memstats_stack_inuse_bytes gauge

go_memstats_stack_inuse_bytes 1.933312e+06

HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.

TYPE go_memstats_stack_sys_bytes gauge

go_memstats_stack_sys_bytes 1.933312e+06

HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.

TYPE go_memstats_sys_bytes gauge

go_memstats_sys_bytes 3.5023996e+07

HELP ingress_controller_config_last_reload_successfull Whether the last configuration reload attemp was successfull.\n Prometheus alert example:\n alert: IngressControllerFailedReload \n expr: ingress_controller_config_last_reload_successfull == 0\n for: 10m

TYPE ingress_controller_config_last_reload_successfull gauge

ingress_controller_config_last_reload_successfull 1

HELP ingress_controller_config_last_reload_successfull_timestamp_seconds Timestamp of the last successfull configuration reload.

TYPE ingress_controller_config_last_reload_successfull_timestamp_seconds gauge

ingress_controller_config_last_reload_successfull_timestamp_seconds 1.52792751e+09

HELP ingress_controller_success DEPRECATED: use ingress_controller_config_last_reload_successfull_timestamp_seconds or ingress_controller_config_last_reload_successfull instead.\n Cumulative number of Ingress controller reload operations

TYPE ingress_controller_success counter

ingress_controller_success{count="reloads"} 91

HELP num_procs number of processes

TYPE num_procs gauge

num_procs 7

HELP oldest_start_time_seconds start time in seconds since 1970/01/01

TYPE oldest_start_time_seconds gauge

oldest_start_time_seconds 1.527800405e+09

HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.

TYPE process_cpu_seconds_total counter

process_cpu_seconds_total 1670.85

HELP process_max_fds Maximum number of open file descriptors.

TYPE process_max_fds gauge

process_max_fds 1.048576e+06

HELP process_open_fds Number of open file descriptors.

TYPE process_open_fds gauge

process_open_fds 26

HELP process_resident_memory_bytes Resident memory size in bytes.

TYPE process_resident_memory_bytes gauge

process_resident_memory_bytes 2.9581312e+07

HELP process_start_time_seconds Start time of the process since unix epoch in seconds.

TYPE process_start_time_seconds gauge

process_start_time_seconds 1.527800406e+09

HELP process_virtual_memory_bytes Virtual memory size in bytes.

TYPE process_virtual_memory_bytes gauge

process_virtual_memory_bytes 8.33855488e+08

HELP read_bytes_total number of bytes read

TYPE read_bytes_total counter

read_bytes_total 3.0748672e+07

HELP resident_memory_bytes number of bytes of memory in use

TYPE resident_memory_bytes gauge

resident_memory_bytes 7.602176e+07

HELP virtual_memory_bytes number of bytes of memory in use

TYPE virtual_memory_bytes gauge

virtual_memory_bytes 9.71493376e+08

HELP write_bytes_total number of bytes written

TYPE write_bytes_total counter

write_bytes_total 2.236416e+06
etricsginx-ingress-controller-7d47954db-45g6g:/etc/nginx# curl localhost:10254/me

HELP cpu_seconds_total Cpu usage in seconds

TYPE cpu_seconds_total counter

cpu_seconds_total 1659.1599999999999

HELP go_gc_duration_seconds A summary of the GC invocation durations.

TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 0.000274582
go_gc_duration_seconds{quantile="0.25"} 0.000437446
go_gc_duration_seconds{quantile="0.5"} 0.000595938
go_gc_duration_seconds{quantile="0.75"} 0.000793277
go_gc_duration_seconds{quantile="1"} 0.025237446
go_gc_duration_seconds_sum 1.117934362
go_gc_duration_seconds_count 1097

HELP go_goroutines Number of goroutines that currently exist.

TYPE go_goroutines gauge

go_goroutines 83

HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.

TYPE go_memstats_alloc_bytes gauge

go_memstats_alloc_bytes 3.870976e+06

HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.

TYPE go_memstats_alloc_bytes_total counter

go_memstats_alloc_bytes_total 9.80505328e+08

HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.

TYPE go_memstats_buck_hash_sys_bytes gauge

go_memstats_buck_hash_sys_bytes 787049

HELP go_memstats_frees_total Total number of frees.

TYPE go_memstats_frees_total counter

go_memstats_frees_total 1.2028919e+07

HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.

TYPE go_memstats_gc_sys_bytes gauge

go_memstats_gc_sys_bytes 2.0199424e+07

HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.

TYPE go_memstats_heap_alloc_bytes gauge

go_memstats_heap_alloc_bytes 3.870976e+06

HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.

TYPE go_memstats_heap_idle_bytes gauge

go_memstats_heap_idle_bytes 4.685824e+06

HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.

TYPE go_memstats_heap_inuse_bytes gauge

go_memstats_heap_inuse_bytes 5.963776e+06

HELP go_memstats_heap_objects Number of allocated objects.

TYPE go_memstats_heap_objects gauge

go_memstats_heap_objects 29009

HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.

TYPE go_memstats_heap_released_bytes_total counter

go_memstats_heap_released_bytes_total 573440

HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.

TYPE go_memstats_heap_sys_bytes gauge

go_memstats_heap_sys_bytes 1.06496e+07

HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.

TYPE go_memstats_last_gc_time_seconds gauge

go_memstats_last_gc_time_seconds 1.5279277822351248e+09

HELP go_memstats_lookups_total Total number of pointer lookups.

TYPE go_memstats_lookups_total counter

go_memstats_lookups_total 13370

HELP go_memstats_mallocs_total Total number of mallocs.

TYPE go_memstats_mallocs_total counter

go_memstats_mallocs_total 1.2057928e+07

HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.

TYPE go_memstats_mcache_inuse_bytes gauge

go_memstats_mcache_inuse_bytes 3472

HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.

TYPE go_memstats_mcache_sys_bytes gauge

go_memstats_mcache_sys_bytes 16384

HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.

TYPE go_memstats_mspan_inuse_bytes gauge

go_memstats_mspan_inuse_bytes 46092

HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.

TYPE go_memstats_mspan_sys_bytes gauge

go_memstats_mspan_sys_bytes 65536

HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.

TYPE go_memstats_next_gc_bytes gauge

go_memstats_next_gc_bytes 7.09296e+06

HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.

TYPE go_memstats_other_sys_bytes gauge

go_memstats_other_sys_bytes 1.372691e+06

HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.

TYPE go_memstats_stack_inuse_bytes gauge

go_memstats_stack_inuse_bytes 1.933312e+06

HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.

TYPE go_memstats_stack_sys_bytes gauge

go_memstats_stack_sys_bytes 1.933312e+06

HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.

TYPE go_memstats_sys_bytes gauge

go_memstats_sys_bytes 3.5023996e+07

HELP ingress_controller_config_last_reload_successfull Whether the last configuration reload attemp was successfull.\n Prometheus alert example:\n alert: IngressControllerFailedReload \n expr: ingress_controller_config_last_reload_successfull == 0\n for: 10m

TYPE ingress_controller_config_last_reload_successfull gauge

ingress_controller_config_last_reload_successfull 1

HELP ingress_controller_config_last_reload_successfull_timestamp_seconds Timestamp of the last successfull configuration reload.

TYPE ingress_controller_config_last_reload_successfull_timestamp_seconds gauge

ingress_controller_config_last_reload_successfull_timestamp_seconds 1.52792751e+09

HELP ingress_controller_success DEPRECATED: use ingress_controller_config_last_reload_successfull_timestamp_seconds or ingress_controller_config_last_reload_successfull instead.\n Cumulative number of Ingress controller reload operations

TYPE ingress_controller_success counter

ingress_controller_success{count="reloads"} 91

HELP num_procs number of processes

TYPE num_procs gauge

num_procs 7

HELP oldest_start_time_seconds start time in seconds since 1970/01/01

TYPE oldest_start_time_seconds gauge

oldest_start_time_seconds 1.527800405e+09

HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.

TYPE process_cpu_seconds_total counter

process_cpu_seconds_total 1670.91

HELP process_max_fds Maximum number of open file descriptors.

TYPE process_max_fds gauge

process_max_fds 1.048576e+06

HELP process_open_fds Number of open file descriptors.

TYPE process_open_fds gauge

process_open_fds 26

HELP process_resident_memory_bytes Resident memory size in bytes.

TYPE process_resident_memory_bytes gauge

process_resident_memory_bytes 2.9581312e+07

HELP process_start_time_seconds Start time of the process since unix epoch in seconds.

TYPE process_start_time_seconds gauge

process_start_time_seconds 1.527800406e+09

HELP process_virtual_memory_bytes Virtual memory size in bytes.

TYPE process_virtual_memory_bytes gauge

process_virtual_memory_bytes 8.33855488e+08

HELP read_bytes_total number of bytes read

TYPE read_bytes_total counter

read_bytes_total 3.0748672e+07

HELP resident_memory_bytes number of bytes of memory in use

TYPE resident_memory_bytes gauge

resident_memory_bytes 7.5976704e+07

HELP virtual_memory_bytes number of bytes of memory in use

TYPE virtual_memory_bytes gauge

virtual_memory_bytes 9.71497472e+08

HELP write_bytes_total number of bytes written

TYPE write_bytes_total counter

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/ingress-nginx/issues/2359#issuecomment-394092303,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFbljD3iKuZGTUpSfUwGGTD1ebQtvIzuks5t4qRmgaJpZM4TYB-5
.

>

Giancarlo Rubio

[toolboc]

@gianrubio,

vts metrics are enabled and loaded :

Fwiw, I am running an ARM build of ingress-nginx based on current master

[aledbf]

@toolboc are you using the dynamic configuration flag? If that's the case there is no prometheus metrics (yet)

[toolboc]

@aledbf

I am not using --enable-dynamic-configuration so it should be disabled by default:

,

[toolboc]

@aledbf

Perhaps I have something misconfigured or there is something else going on, I am noticing the following appears in the logs after enabling vts:

vts.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx vts (unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status/format/json: EOF

[aledbf]

@toolboc please use kubectl exec <ingress pod> curl -v http://0.0.0.0:18080/nginx_status/format/json to see the response from the vts module

[toolboc]

@aledbf

No response, it is as if there is no service being created on port 18080:

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 18080 (#0)
> GET /nginx_status/format/json HTTP/1.1
> Host: localhost:18080
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

[desaintmartin]

@aledbf @gianrubio there is already this dashboard for grafana: https://grafana.com/dashboards/3050 but I haven't found / wrote Prometheus alerts.

[Sexual]

@aledbf I'm experiencing the same issue but when running the command I get the JSON response back.

[aledbf]

Closing. Fixed in #2771