Micrometer: Datadog: units missing from metadata

[fkrull]

Since updating to 1.4, sending units to Datadog doesn't work any more. Metrics are being sent as expected, but they only show up as unitless numbers. Certain metrics sent by e.g. JvmMemoryMetrics still show a unit, but none of our custom metrics do. Downgrading to 1.3.7 with no further code changes fixes the issue.

I did some debugging and came to the conclusion that the units are properly piped all the way to the Datadog API calls; however, Datadog appears to simply ignore them. I'm suspecting PR #1864 which moved the units from the "edit metrics" endpoint to the intake endpoint. Interestingly, Datadog's own API docs don't mention being able to send the unit as part of that request, though I wouldn't be shocked to learn that their API docs are incorrect.

Note: we're using app.datadoghq.eu in case that makes any difference.

[shakuzen]

Pinging @gzussa and team. Any insight on this?

[gzussa]

@fkrull Would you mind to open a support case with our support team so we can have more informations about your org. You can link this issue and mention that i asked you to open the ticket.
https://www.datadoghq.com/fr/support/ Thanks

[gzussa]

We tested our APIs and it is working all fine.
You can try yourself with your environment with following commands:

currenttime=$(date +%s)
curl  -X POST -H "Content-type: application/json" \
-d "{ \"series\" :
         [{\"metric\":\"test.eu.metric.unit\",
          \"points\":[[$currenttime, 20]],
          \"type\":\"gauge\",
          \"interval\": 20,
          \"unit\":\"lock\",
          \"host\":\"test.example.com\",
          \"tags\":[\"environment:test\"]}
        ]
}" \
"https://api.datadoghq.eu/api/v1/series?api_key=XXX"
{"status": "ok"}%

Then go to the metric summary page /metric/summary and you should see test.eu.metric.unit with all metadatas set properly.

PS: We are working on fixing the doc.

[fkrull]

We've mostly been ignoring this problem by staying on 1.3 for now, but I tried a few more things and it seems we can get rid of the problem by disabling descriptions (i.e. disabling them in Micrometer config). This matches the observation that posting test metrics with curl works just fine since the metadata endpoint is never hit for those. My observations when messing around with our codebase were:

• a metric created with 1.3 with descriptions enabled has a unit
• a metric created with 1.5 (different name, same code) with descriptions enabled has no unit
• a metric created with 1.5 (different name again) with descriptions enabled has a unit

Now, if I had to guess (I don't have to, I know), I'd say hitting the metadata endpoint with no unit set overwrites the unit that's sent to the intake endpoint.

[therve]

Hello @fkrull , I believe the attached PR would fix the issue. Would you mind having a look? Thanks.

[shakuzen]

@fkrull thank you for following up and the investigation on your part. Would you be willing to check that the issue is fixed with the latest 1.5.3-SNAPSHOT containing the fix from @therve?

[fkrull]

@shakuzen That seems to have fixed it. :+1: