Apm: How to dynamically add new fields per agent?

[simitt]

Up until 7.0 agent developers had the chance to send new, preliminary or simply agent specific fields within the context object to the APM Server and those data would be stored in _source, but not indexed, in ES.

With Elastic Common Schema (ECS) v1.0.0 the APM Server team moved quite some fields around on the ES schema to be ECS compliant. After applying ECS a couple of fields that are not indexed were left under the namespace context. The namespace context was a little confusing now that a lot of context information moved to its own namespaces, so the team decided to get rid of context on ES level and move all fields under its appropriate root keys. See full list of changes.

context would now only contain information additionally sent from agents, but not being defined on Intake API level. After a few discussions, this information will not be stored any longer, from 7.0 on. As an alternative this issue is opened to discuss other approaches for sending up new or agent specific information in a flexible way. The discussion should at least involve @elastic/apm-agent-devs @elastic/apm-server and @elastic/apm-ui.

[eyalkoren]

Ideally, agent would be able to send new data in a dedicated namespace to be immediately available at UI.
An initial high-level proposal: we all agree on a namespace that will be reserved to these kinds of new data. We will have 4 fields for sending data as key-value pairs:

1. boolean value
2. numeric value
3. string value- to be stored as keyword
4. string value- to be stored as text

Server will save the data accordingly and UI will show all such data in a dedicated location (I don't propose where, will leave that to others that know better).

Can something like that work?

[jalvz]

FWIW, we already have fields defined that are only used by specific agents. Is there a strong need for dynamic mapped fields considering the releases cycles we have currently? Can we simply add defined fields on a need by need basis?

Dynamically mapped fields can be a complication eg. for migrations.

[eyalkoren]

Even with frequent release cycles, and assuming server can add new fields real quick, this means depending on UI to work around a design of where adding every little thing. Another problem is that it means waiting for next stack release and upgrade to get upgraded agent functionalities, instead of just upgrading the agent.
I believe there IS a valid need for that and I assume we (agent devs) will just avoid collecting stuff sometimes because it makes each such addition very complicated.

[jalvz]

this means depending on UI to work around a design of where adding every little thing

not sure i understand how that relates to dynamic mapping?

waiting for next stack release and upgrade to get upgraded agent functionalities

sure, if it out-weights the drawbacks, then its fair.

[sqren]

We currently have context.custom, which might move to custom (or deleted entirely, not sure).
If we keep custom dynamically mapped, agents can freely send whatever data they want, and we can make it show up in a "Custom" tab.

This seems neat, but not super user friendly. I'm afraid it'll be a mixed bag of different things, and hard for the user to understand what's there and why. If this data is useful we probably want to move them to a "proper" location, which will require agents, apm-server (and maybe UI*) to coordinate anyway. I'm therefore not sure of the value of dynamically mapped fields.

* If the new property is moved to an existing location, eg. "http" the UI will automatically display it, without needing any changes.

[formgeist]

custom IMO should be only for the user to decide what's added. I think @eyalkoren's intention is to have the agent send up new data, but not under the "user" umbrella, but simply as the agent.

[felixbarny]

I believe there IS a valid need for that and I assume we (agent devs) will just avoid collecting stuff sometimes because it makes each such addition very complicated.

Even the current workflow requires to align with other agents before sending additional data. Forcing alignment and agreement from other agent owners is important as otherwise we might never consolidate on these dynamic/ad-hoc fields. Examples from the past have also shown that it's important to discuss about adding fields: https://github.com/elastic/apm/issues/33.

The change that the APM Server does not store unknown context fields anymore does actually not change anything in the workflow of the agent devs for adding fields. The only difference is that users will have to wait for the next stack release until they can see the additional properties.

Even with frequent release cycles, and assuming server can add new fields real quick, this means depending on UI to work around a design of where adding every little thing.

From the user's perspective, it's probably not a big difference that they have to update Kibana if they have to update the APM Server anyway. The changes required in Kibana are probably even simpler than in the APM Server so that it's feasible for agent and server devs to provide a PR to Kibana. I'm not sure how it's currently implemented but we could also make it so that all properties under a certain context, like db are displayed in Kibana, without having to specify all sub-fields, like db.statement, db.type, ...

[sqren]

we could also make it so that all properties under a certain context, like db are displayed in Kibana, without having to specify all sub-fields, like db.statement, db.type,

That's how it works today :)
We have a config which decides which top-level properties we want to display as tabs, and show everything beneath them in a table. Mostly the child properties are sorted alphabetically, but we have the option to specify a custom order:
https://github.com/elastic/kibana/blob/a6291afd7d42b380f7e08c0e1de5d65d735d890e/x-pack/plugins/apm/public/components/shared/PropertiesTable/propertyConfig.ts#L30-L86

[eyalkoren]

@felixbarny I raised two separate concerns- one is that addition of anything requires a tedious process and while aligning between agents can be done quite quickly, making the process include all components is a different story. The other is the wait for stack releases.
@sqren so for adding to DB, for example, user will have to configure to add db as a tab? And, are new fields agent sends under db will be stored?

[sqren]

so for adding to DB, for example, user will have to configure to add db as a tab? And, are new fields agent sends under db will be stored?

The only data we are showing dynamically are those listed in the propertyConfig. So that is url, http, host, service, process, user and labels.
We show the sub properties for these properties for the transaction and errors samples:

Since db is not part of this list, properties under db will not show here (or anywhere else) except when explicitly added.

user will have to configure to add db as a tab

No, users cannot configure which tabs show up.

[simitt]

@sqren span.db has been nested under context so far, and will be moved from 7.0 on. Let's ensure we adapt the UI to the ECS changes, but I suggest a different thread for this.

@eyalkoren new fields agents send up under db would still not be shown, as we are not storing undefined fields.

[Qard]

The custom context is currently used by the Node.js agent for storing lambda information, so we should make sure there's a lambda object at the top-level if we want to eliminate that use.

https://github.com/elastic/apm-agent-nodejs/blob/master/lib/agent.js#L413-L427

[simitt]

custom.context will _NOT_ be eliminated, as it is defined on the API level. It will be moved to transaction.custom and error.custom though on ES level.
However, afaik custom should only be used by users and not by agents themselves.

[jalvz]

afaik custom should only be used by users and not by agents themselves

is not tags the one that is for the users?

(i think the lambda field illustrates @sqren point: we rather define each field purposefully and sync agents-server-ui, even if that means slowing down; than having a meaningless placeholder for all kind of disparate things that might not have anything to do with each other)

[formgeist]

is not tags the one that is for the users?

Tags if they want them indexed, custom for anything else (still only for users). This was always the intention to have spaces for the users only, so no cluttering it with agent data.

[roncohen]

there are two issues at play here:

1. how to allow agents to add new fields (e.g. db.statement) that are collected automatically
2. how do we let users add custom data that doesn't fit into tags/labels

In this issue, @simitt brought up the point of how agent devs should add new fields (1). We have a process for that as defined here. Let's discuss (2) separately.

As Felix mentioned, the process itself doesn't need to change with this, although some of the argumentation in it does. I've submitted a PR to update it here.

Specifically, this part of the process doesn't change:

Agent devs can release new versions that send the new fields as soon the changes are merged into APM Server.

This forcing function that ensures that we are aligned is critical. Allowing agents to send up agent-defined fields (as opposed to user defined fields) before the changes are in APM Server takes away that forcing function of alignment. This is something we all agreed on already and it doesn't change with changes that @simitt brought up.

The proposal that @eyalkoren brought up mentioned that

Server will save the data accordingly and UI will show all such data in a dedicated location (I don't propose where, will leave that to others that know better).

The point is that we don't want to have an area in the UI that is a dumping ground for arbitrary fields. I think it's really important to have a conversation around where every new field go. We need to have a decision per new field/group of new fields, otherwise we'll end up with a big mess. The forcing function described is supposed to ensure that doesn't happen.

Unless we think this forcing function is no longer important, I think we should table this issue.

[graphaelli]

The process has been moved into an issue template in this repo thanks to @basepi in https://github.com/elastic/apm/pull/263.

https://github.com/elastic/apm-server/pull/1961 added an experimental mode shortly after this issue was last discussed, it at least partially addresses the need here.

Closing this out as I don't see any more action needed but feel free to reopen or take specific concerns to a new issue.