Micrometer: Support for Dynatrace metrics API v2

[obarcelonap]

Dynatrace has released a new metric ingestion API much flexible than previous one.
https://www.dynatrace.com/support/help/dynatrace-api/environment-api/metric-v2/post-ingest-metrics/

Ingestion flow has changed significantly:

• no need to preregister metrics
• metrics are not constrained to a custom device anymore
• more flexible naming conventions (eg. no custom: prefix)
• payload type as line protocol
• possibility to ingest gauges and counters

Since there are quite changes and new api is still in EAP, I'd propose to create a new shinny meter registry, let's say micrometer-registry-dynatrace2 and deprecate the first versionmicrometer-registry-dynatrace so first version would be still available to be used to do not lose the history of the custom metrics registered with v1.

If is ok from your side, I'd try to contribute in this new registry in the upcoming days.

[shakuzen]

We would probably want the API to be GA before we merge and release it, but if you want to work on a pull request in the meantime, that's fine. Is there a timeline for the API becoming stable?

[jmcolyerkr]

@obarcelonap Have you made any progress on this? v1 has some limitations that v2 solves. The major issue we have is having duplicate keys names with different dimensions. In v1 this results in a "400 Constraint Violation" where v2 handles it as expected. This makes v1 unusable for our use case.

[jmcolyerkr]

@obarcelonap I opened a support ticket with the Dynatrace team regarding the GA for the v2. Here's the reponse...

"The ingest endpoint is already stable at this point and the official version of v2 is planned to be available some time at the end of Q1/ 2021."

On a side note I was able to find your fork with the work you did for the new api. I had one concern regarding creating a dynatrace2 version. This would require some coordination with the spring-boot team to change/create a new DynatraceMetricsExportAutoConfiguration as the package names will be different.

[obarcelonap]

@jmcolyerkr I pushed the progress into a branch in my repo, branch is working and reports gauges and counters correctly. It is pending to stabilize it, trying out all the samples and possible corner cases.
https://github.com/obarcelonap/micrometer/tree/add-dyntrace-meter-registry-for-metrics-api-v2-squashed

Regarding creating a new implementation module for the 2nd version, the idea is to have a clean start for this new version as I think it has less complexity. Also because we don't need to manage compatibility with v1 - using same implementation and having a prop with the version could create problems. For example, defaulting to v2, may end with new metrics keys for the same metrics, causing confusion to the users. Or defaulting to v1 would mean users may need to realize there is other impl and manually change it which I think is against micrometer/spring boot philosophy to provide good defaults.
So I think create a new impl is easier to manage in terms of supportability and there is another implementation did the same.
But in any case, it is easy to change, we can jus put a proxy on top of both impl and decide what to use based on some prop (or something else).
Maybe maintainers can help us a bit here in take the decision. @shakuzen what to do you think on this matter?

[jmcolyerkr]

@obarcelonap thanks for your work putting the new library together! I will build it locally and test it in our environment and report back.

[jmcolyerkr]

@obarcelonap Reporting back...

These 3 properties are now missing deviceId, group, technologyType. DeviceId I believe is a good dimension that is added and makes it easy to find the metrics under custom devices.

Registry Timer doesn't appear to be sent...
Timer timer = registry.timer(“my.timer", "class", className, "method", methodName); timer.record((totalTimeMillis), TimeUnit.MILLISECONDS);

my.timer does not show up in the POST body.

Counter successCounter = registry.counter("my.count", "class", className, "method", methodName); successCounter.increment();

my.count does show up in the POST body.

There are a few errors which seems to be case sensitivity and possibly not quoted properly.

{"linesOk":163,"linesInvalid":29,"error":{"code":400,"message":"First 10 errors reported","invalidLines":[{"line":3,"error":"failed to parse metric key of message 'system.load.average.1m 2.33154 1608218155858'; error at '1' (index 20)"},{"line":8,"error":"invalid dimension key in message 'cache.size,cache=myCache,cacheM'; error at 'M' (index 63)"},{"line":14,"error":"invalid dimension key in message 'cache.size,cache=myCache2,cacheManager=cacheManager,n'; error at 'M' (index 42)"},{"line":20,"error":"invalid metric type in message 'jvm.memory.max,area=nonheap,id=Compressed Class Space 1073741824'; error at 'C' (index 42)"},{"line":24,"error":"invalid metric type in message 'jvm.memory.used,area=heap,id=G1 Eden Space 130023424 16082181558'; error at 'E' (index 32)"},{"line":26,"error":"invalid metric type in message 'jvm.memory.max,area=nonheap,id=CodeHeap 'non-profiled nmethods' '; error at ''' (index 40)"},{"line":27,"error":"invalid metric type in message 'jvm.memory.used,area=heap,id=G1 Survivor Space 13631488 16082181'; error at 'S' (index 32)"},{"line":28,"error":"invalid dimension key in message 'cache.size,cache=myCache3,cacheManager=cacheMana'; error at 'M' (index 47)"},{"line":32,"error":"invalid dimension key in message 'cache.size,cache=myCache4,cacheManager=cacheManager,na'; error at 'M' (index 41)"},{"line":33,"error":"invalid metric type in message 'jvm.memory.max,area=heap,id=G1 Old Gen 4294967296 1608218155860'; error at 'O' (index 31)"}]}}

[damass]

@jmcolyerkr Thank you for testing things out. I'm helping @obarcelonap with this.

With the deviceId, group & technologyType in the configuration, these were all needed in the v1 API because each metric had to be assigned to a custom device. However with the v2 API this is not the case as metrics can now be free of any entity. This means that now a custom device isn't needed for metrics to be available in Dynatrace. There is a new reserved dimension which is used to assign the metrics to an existing entity so if there is the OneAgent on the host we can now map these metrics directly to that process or host. From the documentation :

Dynatrace reserved dimensions
The dt.entity. key is a Dynatrace reserved dimension key that relates the metric to the monitored entity provided as dimension value (for example, the dt.entity.host=HOST-06F288EE2A930951 dimension maps the data points to the host with the ID of HOST-06F288EE2A930951).

With this new properties could be added to specify the HOST ID or the Process Group Instance ID and append one of those values to the metric.

As for the timer's I don't believe any timer meter type has been enabled, only gauge and counter types. At the moment I'm not sure why your counter isn't showing up, but I'll have a look.

Finally, there are some known issues with the quoting that I'm working on now, but also looks like the dimensions need to be lowercase so I'll look into that as well. Again from the docs :

Dimensions are specified as key="value" pairs. You can specify up to 50 dimensions, separated by commas (,).
Allowed characters for the key are lowercase letters, numbers, hyphens (-), and underscores (_). Special letters (like ö) are not allowed. The key can be in the key.key-section format.

[jmcolyerkr]

@damass Host Id or Instance Id wouldn't be good for the aggregation I need. For example, if i have 4 instances I want to know the HTTP failure rate for all 4. I could add a common tag for "application", but deviceId is what I was using. Also, if you look in Dynatrace -> Technologies -> Custom Devices the metrics would show up under the deviceId given. IMO it should be an optional property.

The timer is a must for me as that's the main reason for me pulling in this library. Long story short I use it to create a dashboard which shows performance at class/method level. I know you can drill in but I get a nice easy dashboard.

Feel free to reach out to me and I can do additional testing and/or drop a few lines of code.

EDIT - my instance id's will be dynamic and change often, so I can't "hard code" group them.

[damass]

@jmcolyerkr I've done some updates and now all the different Meter Types should be supported. Also, I've added some additional properties to help with what we were talking about. There are now deviceName, groupName, and entityId properties which should cover all the different use cases.

Please note that the deviceName and groupName are just added dimensions to the metric and will NOT attach the metric to an existing entity.

Since you may already have an existing Custom Device for these metrics you can use the entityId property and enter the CUSTOM_DEVICE- and the metrics will then be attached to it. The currently supported entity types are HOST, PROCESS_GROUP, PROCESS_GROUP_INSTANCE, CUSTOM_DEVICE, and CUSTOM_DEVICE_GROUP.

All of the new properties as well as the other dimensions will be available in the new metrics UI screen: https://{your-environment-id}.live.dynatrace.com/ui/metrics

[jmcolyerkr]

Is there anything preventing these pull requests from being merged? I thought there was a wait for V2 to be officially GA's but I believe that it has been. Let me know how I can help.

[damass]

@shakuzen, would you be the one to review the PR for this issue? Please let me know if there is anything that needs to be changed or anyone else to loop in.

[jpandres]

Hi there, thanks for all the open source contributors here! I just stopped here to say that as a user of Dynatrace, I would really really like this library to be released to start using it from our applications. 🙏

[jonatan-ivanov]

@jpandres You can use it right now with the V1 API: https://micrometer.io/docs/registry/dynatrace

[jpandres]

@jpandres You can use it right now with the V1 API: https://micrometer.io/docs/registry/dynatrace

Hi Jonatan, thanks for the suggestion. I am using it (or trying to, because I have issues when going through Active Gates), but API v2 is way less restrictive and would prefer to use it when this library becomes available.

[jonatan-ivanov]

Closing this since the feature was merged: https://github.com/micrometer-metrics/micrometer/pull/2619
Thanks everyone for the contributions! :)

[shakuzen]

Users who have been looking forward to this feature, please try out 1.8 snapshots and let us know any feedback on it.