Caddy: Metrics clashing name

[M0NsTeRRR]

I'm running exporter-exporter to expose caddy metric with TLS and cert authentication and it appears go_memstats_alloc_bytes
is in collision with an other metric

déc. 27 19:24:31 reverse-proxy exporter_exporter[43467]: time="2020-12-27T19:24:31+01:00" level=error msg="Verification for module 'caddy' failed: Failed to decode metrics from proxied server: text format parsing error in line 343: second HELP line for metric name \"go_memstats_alloc_bytes\""

The two metrics

# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.058576e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.4330096e+07

If I try to parse them with the prometheus client parser

$ python3 -c 'import sys; from prometheus_client.openmetrics import parser; list(parser.text_string_to_metric_families(sys.stdin.buffer.read().decode("utf-8")))'
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.058576e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.4330096e+07

This is the output I get

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/openmetrics/parser.py", line 24, in text_string_to_metric_families
    for metric_family in text_fd_to_metric_families(StringIO.StringIO(text)):
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/openmetrics/parser.py", line 615, in text_fd_to_metric_families
    yield build_metric(name, documentation, typ, unit, samples)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/openmetrics/parser.py", line 479, in build_metric
    raise ValueError("Clashing name: " + name + suffix)
ValueError: Clashing name: go_memstats_alloc_bytes_total

Openmetrics suffixes
https://github.com/prometheus/client_python/blob/f81891f57bf3d20a9bb87ea62b6c6b4a6c06f803/prometheus_client/openmetrics/parser.py#L466

type_suffixes = {
        'counter': ['_total', '_created'],
        'summary': ['', '_count', '_sum', '_created'],
        'histogram': ['_count', '_sum', '_bucket', '_created'],
        'gaugehistogram': ['_gcount', '_gsum', '_bucket'],
        'info': ['_info'],
    }

[hairyhenderson]

Hi @M0NsTeRRR, I'm not sure what the issue is here - these are both metrics exposed by the Prometheus go client - https://github.com/prometheus/client_golang. They are named differently, and so by classical Prometheus rules they don't conflict, though I do recognize that OpenMetrics has some _slightly_ different expectations.

To be clear, when you parse with the python client, are you parsing directly or through the exporter-exporter? If the latter, try taking that out of the equation and scraping metrics directly.

In the Caddy codebase we enable OpenMetrics negotiation automatically, which may be why the python client is running into that issue.

[M0NsTeRRR]

Hello @hairyhenderson,

When I parse them with the python client I use the direct output of the caddy exporter.

I've read this issue : https://github.com/prometheus/common/issues/253#issuecomment-688499208
and it's the same case as here, if I'm not mistaken.

[hairyhenderson]

Thanks for finding that issue @M0NsTeRRR. It does look significantly related. I think what that means is the presence of go_memstats_alloc_bytes_total is a bug in prometheus/client_golang since it causes the client to be non-conformant with OpenMetrics.

I'm not sure what we can do other than disabling OpenMetrics negotiation support or - probably better - make it configurable.

It'd probably also be a good idea to file a bug in prometheus/client_golang, since it's producing this output.

Modulo holidays/etc, I can do both of these things at some point over the next few days 😉...

[M0NsTeRRR]

Thanks for finding that issue @M0NsTeRRR. It does look significantly related. I think what that means is the presence of go_memstats_alloc_bytes_total is a bug in prometheus/client_golang since it causes the client to be non-conformant with OpenMetrics.

I'm not sure what we can do other than disabling OpenMetrics negotiation support or - probably better - make it configurable.

It'd probably also be a good idea to file a bug in prometheus/client_golang, since it's producing this output.

Modulo holidays/etc, I can do both of these things at some point over the next few days 😉...

Of course thanks you, i'm learning golang since few days so I can't help you a lot on that works ahah :), feel free to tag me here or in the new issue if you need some logs or config files

[hairyhenderson]

Ok, I've added an option to disable the negotiation in #3944, and logged https://github.com/prometheus/client_golang/issues/829.

Once #3944 is merged this issue can be closed, since you'll be able to configure the metrics handler to avoid the OM negotiation, which should be fine.

It may be too late to get this into Caddy 2.3.0, but hopefully not 😅

[hairyhenderson]

@M0NsTeRRR when 2.3.0 is released (probably sometime over the next week) you can upgrade to that and set disable_openmetrics.

If you're using the metrics handler that's on the admin endpoint you'll need to switch to the explicit metrics handler instead since the admin one can't be configured.

Based on conversation in the prometheus/client_golang issue there likely is a bug there where ambiguous metrics are being exposed.

[M0NsTeRRR]

Okay, thanks you @hairyhenderson for your quick fix :)