For a bunch of users, it would be nice to be able to label/tag all the data coming from specific agents. You might want to label APM data with the business unit that is responsible, the name of the rack where the agent is running or whatever it might be.
We already have amazing high cardinality support in Elasticsearch so letting people add more shouldn't be a problem.
To begin with, it would be great if all the agents supported a way to add static labels when starting the agent, either through env vars, code configuration or command line arguments. Each agent will have a different perspective on this, which is fine, as long as we can align on the name for the env var ;)
The data should be sent up as tags to the intake API (until this gets merged) and I think we should start by just setting it on all transactions, errors and metricsets.
This fits nicely into "metadata" in the intake API. It requires some work from the server and we _could_ start without it just by having the agents add the tags to the events. Happy to hear thoughts on this.
As an aside, it would be pretty great if central config could apply configurations to agents based on, amongst other things, their static labels.
roncohen
All 26 comments
I have this in the Ruby agent called default_tags: https://github.com/elastic/apm-agent-ruby/blob/master/docs/configuration.asciidoc#default_tags
mikker
on 13 Feb 2019
Each agent will have a different perspective on this, which is fine, as long as we can align on the name for the env var ;)
ELASTIC_APM_SERVICE_LABELS?
This fits nicely into "metadata" in the intake API. It requires some work from the server and we _could_ start without it just by having the agents add the tags to the events. Happy to hear thoughts on this.
If we do add it to events, then for backwards compatibility the agents would have to keep adding it to events until their next major release. Unless there's a pressing need, I'd suggest we only set it in stream metadata and tie the feature to a stack release.
axw
on 14 Feb 2019
sounds good to me @axw. I don't know of a common format for specifying multiple key/value pairs in a single environment variable. Unless anyone else is familiar with one, let's use this one?:
ELASTIC_APM_SERVICE_LABELS=key1=val1,key=val2
To summarize, agents can start sending the labels at the metadata level as soon as we're in agreement. APM Server allows agents to send arbitrary data in metadata, but it's not stored. When users upgrade their APM Server the feature will start working if they have an up-to-date agent.
Pinging @elastic/apm-agent-devs for visibility. I'll post the agent checkboxes in a few days unless someone has objections.
roncohen
on 20 Feb 2019
Fancy checkboxes:
| Agent | Yes | No | Indifferent | N/A | Link to agent issue |
| --------|:---------------------:|:--------------------:|:---------------------:|:--------------------:|:---------------------|
| .NET |
- [x]
- [ ]
- [ ]
- [ ]
| Go |
- [x]
- [ ]
- [ ]
- [ ]
| Java |
- [x]
- [ ]
- [ ]
- [ ]
| Node.js |
- [x]
- [ ]
- [ ]
- [ ]
| Python |
- [x]
- [ ]
- [ ]
- [ ]
| Ruby |
- [x]
- [ ]
- [ ]
- [ ]
| RUM |
- [ ]
- [ ]
- [ ]
- [x] Global tags are already supported
roncohen
on 28 Feb 2019
Fine with commas – but want to throw in the url/form data format key=value&other=value for familiarity's sake. @watson had concerns that & can be troublesome in a terminal. Feel free to ignore this.
mikker
on 28 Feb 2019
thanks @mikker. Let's stick to the comma separated list for now. Seems easier to humanly decipher than the & and i agree there could be issues with & on the command line.
roncohen
on 1 Mar 2019
I'm going to assume .NET and Python are on board. @elastic/apm-agent-devs please update table with a link to your own issues
roncohen
on 1 Mar 2019
I was too quick checking the Node.js checkbox. I was just voting on the concept, not the actual name of the config option - for some reason I missed that part of the question.
So a few outstanding questions:
- Did we settle on renaming tags to labels?
- I'm not too keen on the config name "service labels". Something like "global labels" seems much more logical to me, but maybe I'm missing the point?
watson
on 1 Mar 2019
- Did we settle on renaming tags to labels?
Not sure we did but I'm +1 on renaming tags to labels.
- I'm not too keen on the config name "service labels". Something like "global labels" seems much more logical to me, but maybe I'm missing the point?
I'm fine with either.
felixbarny
on 1 Mar 2019
I'm just afraid to ask people to create issues on the agent repos until this have been settled 😅 Otherwise we might end up in a situation similar to the one we got with the environment config option where some agents add support for something that is then later changed or delayed, which at best is just confusing for users, but at worst requires the agent to release a breaking change.
watson
on 1 Mar 2019
@roncohen Python is indeed on board, and I did check the checkbox last week. But it appears that there is a bug with those fancy table'd checkboxes and (I assume) Firefox. I can only make it stick by manually editing the markdown text
beniwohli
on 4 Mar 2019
@beniwohli Not just Firefox. This was mentioned when they were introduced as something we'd just learn to live with.
mikker
on 4 Mar 2019
@watson I suggested ELASTIC_APM_SERVICE_LABELS to fit with ELASTIC_APM_SERVICE_NAME and ELASTIC_APM_SERVICE_VERSION, service-level metadata which are added to each event. Service name and version are also statically defined.
Having said that, it makes it very service-centric, and labels may not be uniformly applied across all units of a service. e.g. the "which rack is it running on" example in the description. So... other ideas?
ELASTIC_APM_LABELSELASTIC_APM_STATIC_LABELSELASTIC_APM_GLOBAL_LABELSELASTIC_APM_DEFAULT_LABELSELASTIC_APM_<yourword>_LABELS
axw
on 5 Mar 2019
I like ELASTIC_APM_GLOBAL_LABELS
felixbarny
on 5 Mar 2019
I'm fine with that. Any naysayers? @elastic/apm-agent-devs @roncohen
Here's what I'd propose for documentation:
|Environment|Default|Example|
|-----------------|----------|-----------|
|ELASTIC_APM_GLOBAL_LABELS||dept=engineering,rack=number8|
Labels added to all events, with the format key=value[,key=value[,...]]. Any labels set by application via the API will override global labels with the same keys.
axw
on 8 Mar 2019
Perhaps add a note on how users can escape commas in values with \, or whatever if we want to suppport that.
Could be someone wanted dept=engineering\,finance\,hr,rack=number8 ¯_(ツ)_/¯
mikker
on 8 Mar 2019
I think it's easier to just disallow , and =. Otherwise stuff can get complicated quickly (for example how to escape the escape character etc). I don't think there is a lot of added value allowing these chars within a label. If we do see that this is an important use-case, we can still add that later without breaking changes.
felixbarny
on 8 Mar 2019
I agree 👍
mikker
on 8 Mar 2019
Looks good to me!
On Fri, 8 Mar 2019 at 12.34, Mikkel Malmberg notifications@github.com
wrote:
I agree 👍
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/apm/issues/48#issuecomment-470899006, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAC6JCPQLvafLTG1276Jfc3xobQUfUUks5vUkqugaJpZM4a5Xs1
.
roncohen
on 8 Mar 2019
fyi, this has just been merged in apm-server
jalvz
on 29 Mar 2019
@elastic/apm-agent-devs Be aware that the agents shouldn't implement support for global labels until an APM Server with support for these is released, which will be v7.1.
When that version lands the agents should document support for global labels as a feature that requires v7.1 of the APM Server in order to work. If a user uses it on an older APM Server, the global labels will just be ignored, but the rest of the metadata event will still be accepted.
The issue is that if the agents just added support for global labels today they would have to manually put the global labels on each and every transaction, error and metricset event. Then when v7.1 was released we'd have to make a switch to put it in metadata instead. This would then result in a loss of data for users not yet upgraded to v7.1 which would be a bad experience. Hence waiting till v7.1 is released seems to be the best approach.
watson
on 29 Mar 2019
Implementation issues exist for all agents, except RUM which already has global tags, and server support has been merged. Closing this now.
axw
on 15 Apr 2019
I think it's easier to just disallow , and =
Can I confirm these characters cannot appear in any configuration? There's no way to escape them?
OrangeDog
on 2 Aug 2019
That's correct. What's your use case?
felixbarny
on 2 Aug 2019
@felix-lessoer
What's your use case?
Integrating agent attachment and configuration with Spring Boot.
When list properties are loaded from YAML, I should verify that people aren't trying to include a literal comma.
OrangeDog
on 2 Aug 2019
Can we add multiple values to one label? Like the one suggested above dept=engineering,finance,hr
So that multiple values are added as an array of departments.
Can be useful for RBAC in the "Grant read privileges to specific documents", so that all 3 departments can have access to this data.
peterdkdp
on 9 Mar 2020
Related issues
alvarolobato
·
9Comments
danielfariati
·
9Comments
mikker
·
11Comments
felixbarny
·
9Comments
jianzzz
·
12Comments
Most helpful comment
I think it's easier to just disallow
,and=. Otherwise stuff can get complicated quickly (for example how to escape the escape character etc). I don't think there is a lot of added value allowing these chars within a label. If we do see that this is an important use-case, we can still add that later without breaking changes.