Apm: RFC: destination service name

[axw]

Is your feature request related to a problem? Please describe.

For the service maps feature, we need to report the logical destination of a span in order to discover the unique connections between services without processing every single span. This includes spans representing interactions with both instrumented and non-instrumented destination services.

Describe the solution you'd like

Extend ECS with destination.service.name, which would hold the name for the destination service. This will hold a single value that the service maps code can use int is composite aggregation, as described in https://github.com/elastic/kibana/pull/42120.

For clustered services, or services with multiple network address, the name should be ideally be the same no matter which service node or address is accessed. This is similar to the expectations of ECS's service.name field. This is not always going to be possible to infer, but we can provide opinionated defaults along with an API to override the value.

Below is an initial set of rules to get the conversation started, based on the "extract_destination" pipeline from https://github.com/elastic/kibana/pull/42120.

HTTP(S)

Capture the scheme, host, and non-default port.

Example 1: capture scheme, host and non-default port; omit userinfo, path, and query

Request URI: "http://user:[email protected]:1234/path?query"
Resulting context destination.service.name: "http://testing.local:1234"

Example 2: omit default port

Request URI: "https://www.elastic.co:443/products/apm"
Resulting context destination.service.name: "https://elastic.co" (default port omitted)

Example 3: capture IPv6 address

Request URI: "http://[::1]/"
Resulting context destination.service.name: "http://[::1]"

Elasticsearch, Redis, Databases

For Elasticsearch, we could in theory identify the destination service using the Elasticsearch cluster name. To do that would require agents to inject a query to capture cluster info. Our agents are passive and should not be doing this, and the client may not necessarily have permissions to access cluster info anyway. Similar stories exist for other clustered databases.

Therefore, we should report the destination service name using the database/case span subtype, e.g.

• destination.service.name: "elasticsearch"
• destination.service.name: "redis"
• destination.service.name: "mysql"
• etc.

It is most typical for applications to target a single instance of any one database, so this should be sufficient in most cases. Agents may introduce an API to enable developers to override the name as needed.

Messaging systems

destination.service.name should contain the message queue/topic name. This should enable proper aggregation when queues/topics are deployed on multiple brokers (eg Kafka partitions).

Describe alternatives you've considered

In the original service maps POC, it was proposed that we address this by using destination.address. There are a couple of issues with this, described in https://github.com/elastic/apm/issues/115#issuecomment-555814374.

[eyalkoren]

@axw Awesome!
I added a proposal for Messaging systems. Please edit as you wish.

[roncohen]

thanks for driving this forward @axw!

In the POC, i used service.address not only for uninstrumented serviced, but actually for all services as a way to discover unique connections between services.

Since the field proposed here will contain an abstract "destination" (queue name for example) which doesn't necessarily correspond to a service _name_, maybe we can come up with a field name that signals that's it's an abstract or logical service destination? Maybe destination.logical or even servicemap.destination (destination for use in the service map only). I admit none of those are particularly good. I'll keep thinking on it :)

[SergeyKleyman]

@axw

where that destination service is not instrumented

Could you please clarify what do you mean by that considering that the source agent doesn't know if the destination service is instrumented?

[axw]

@roncohen

In the POC, i used service.address not only for uninstrumented serviced, but actually for all services as a way to discover unique connections between services.

Thanks, I've fixed the description.

Since the field proposed here will contain an abstract "destination" (queue name for example) which doesn't necessarily correspond to a service name, maybe we can come up with a field name that signals that's it's an abstract or logical service destination? Maybe destination.logical or even servicemap.destination (destination for use in the service map only). I admit none of those are particularly good. I'll keep thinking on it :)

Yeah, the name feels iffy to me too. Some more options:

• destination.name
• destination.logical.name
• destination.resource.name
• destination.entity.name

@SergeyKleyman

where that destination service is not instrumented

Could you please clarify what do you mean by that considering that the source agent doesn't know if the destination service is instrumented?

The statement wasn't about the how, but the why - from the agent's perspective, anyway. I've updated description, let me know if it's still unclear.

[roncohen]

I think we can end up with needing to take the kafka cluster name and put it together with a queue name. So the content of the field will end up being <kafka-cluster-name>/<queue> or similar. So, the .name postfix might not be applicable.

destination.resource or destination.logical seems fine for me?

[axw]

I think we can end up with needing to take the kafka cluster name and put it together with a queue name. So the content of the field will end up being / or similar. So, the .name postfix might not be applicable.

I'd say that's still a name, it's just a composite one.

destination.resource or destination.logical seems fine for me?

I don't have a terribly strong opinion here. If I were to pick one of those it would be destination.resource, on account of it being a noun, making it more natural to refer to.

[webmat]

Thanks for looping me in on this discussion, @roncohen! I like the idea of hammering out the semantics for this. This is more or less the idea I had in mind for service.address.

In other places, like source.address and destination.address, the .address field is meant to capture the address in its original format, when you're unsure what it will look like (IP or domain or unix socket).

But inside service.* we don't have this need. I think it would make sense to use service.address (rather than .name) to give a fine grained idea of what the destination is. In this sense, I really like the http part of the proposal. I think we could expand for databases and queues as well, like @roncohen describes with the <kafka-cluster-name>/<queue> example.

Perhaps it would make sense to fill both fields. Here's an attempt:

HTTP

• destination.service.name: "testing.local"
• destination.service.address: "http://testing.local:1234"

(or both equal to .address?)

Databases

• destination.service.name: "Elasticsearch"

• destination.service.address: "https://d05202253e6c45748075c2752c1bc7d9.us-central1.gcp.cloud.es.io:9243"

• destination.service.name: "mysql"

• destination.service.address: "mysql.example.com:3306"

Queueing Systems

• destination.service.name: "Kafka"
• destination.service.address: "kafka.example.com:9092/queue-name"

Custom field or not?

Now the service field set is not currently marked as being nestable under destination at this time. This is right in line with a much bigger discussion around the tension between:

• should source/destination be exact copies of client/server (the latter only adding semantic meaning) but where both are purely meant to capture details about the network exchange.
• or should we allow the nesting of many things under source/destination (process, service, host, etc.)

For now I can't confidently say on which side we're going to fall, with regard to that bigger discussion. So I'd recommend using a purely custom field, rather than nesting directly inside the existing ECS source and destination field sets.

The custom fields could be apm.destination.service.name and apm.destination.service.address.

Using a custom field will avoid causing confusion on whether this is an ECS field or not.

[roncohen]

thanks @webmat and @axw. OK, since we're not doing an ECS field for now, we have some more flexibility. Later, when assumptions have evolved, we can look at standardizing on ECS.

These three fields will be added to span documents. All should be keywords:

• apm.destination.service.address
  This is the address, with the port (even when it's the default port), without any scheme.
  We'll use it for detecting which uninstrumented services are "the same".
• apm.destination.service.name
  For displaying service names for services which we don't have the actual name (uninstrumented/external)
• apm.destination.service.type
  For displaying icons or similar. This should be equal to the span.type.

The Intake API fields will be:

• context.destination.service.name
• context.destination.service.address
• context.destination.service.type

Using apm.destination.service.address to detect sameness has the disadvantage that if two services use two different destination addresses for the same uninstrumented service, we'll show it as two different services. That's OK for now.

Examples

HTTP(S)

apm.destination.service.address: Capture host, and port.
apm.destination.service.name: Capture the scheme, host, and non-default port.
apm.destination.service.type: Same as span.type

Example 1: capture scheme, host and non-default port; omit userinfo, path, and query

Request URI: "http://user:[email protected]:1234/path?query"
Resulting context:

• context.destination.service.name: "http://testing.local:1234"
• context.destination.service.address: "testing.local:1234" (note: no scheme)
• context.destination.service.type: "external

Example 2: omit default port in name

Request URI: "https://www.elastic.co:443/products/apm"
Resulting context

Resulting context:

• context.destination.service.name: "https://www.elastic.co" (default port omitted)
• context.destination.service.address: "www.elastic.co:443"
• context.destination.service.type: "external

Note: ports should always be in address, even if the request is using the default port.

Example 3: capture IPv6 address

Request URI: "http://[::1]/"
Resulting context:

• context.destination.service.name: "http://[::1]"
• context.destination.service.address: "[::1]"
• context.destination.service.type: "external

Elasticsearch, Redis, Databases, Messaging server

When talking to clusters (more than one node/server), we'll require you to use the same "boostrap list" across your different services to be able to detect they are the same cluster. If multiple hosts are specified, agents need to use the first in the list. For example, see the "bootstrap" setting in this kafka producer.

If a message is sent to a broker and arrives at an instrumented service, we'll likely not show the broker between the two services in the first version of the service map. We'll just show a connection between the two instrumented services. That's OK as a start.

Example 1: Kafka cluster

bootstrap list: "server1,server2"
queue: myqueue

Resulting context:

• context.destination.service.name: "server1"
  use the first host as the name
• context.destination.service.address: "server1:9092/myqueue"
  use the first host, add port, add / and queue name
• context.destination.service.type: "messaging

Note: added default port to address

Example 2: Postgres DB

host: "server1:2424"

Resulting context:

• context.destination.service.name: "server1:2424"
• context.destination.service.address: "server1:2424"
• context.destination.service.type: "db

Future work

In the future, we may want to add subtype to be able to show "kafka" or similar in the UI. Let's wait with that. In general, I consider the name attribute something we can keep iterating on.

Unless big objections come up, i'll update the description and we can call this decided

[eyalkoren]

@roncohen is context.destination.service.address going to be used in the first iteration of the service map? If not, let's add it separately and go with the original suggestion (simple common names for the logical name of clustered services), so that we don't all have to modify if we decide something else in the future. There is a big difference in implementation for the agents between the original suggestion and the last one, which is fine, but better commit to this effort when finalized.
Hopefully we can come up with a way to find the logical destination name without enforcing limitations such as:

When talking to clusters (more than one node/server), we'll require you to use the same "boostrap list" across your different services to be able to detect they are the same cluster.

Also:

I think we can end up with needing to take the kafka cluster name and put it together with a queue name.

The concern is that the same topic name is used within two different Kafka clusters in the same app?

If a message is sent to a broker and arrives at an instrumented service, we'll likely not show the broker between the two services in the first version of the service map. We'll just show a connection between the two instrumented services.

Why? Because of disability to infer the same logical name in both ends? If so, we can deliver the computed logical name as a Message/Record metadata, alongside the trace context we pass.

[roncohen]

caught up with @eyalkoren. Turns out that getting the creation/initial destination address can be problematic because we might only attach after client creation. We'll have to go with simple names for now. I'll create a new issue to ensure there's no confusion.

[roncohen]

replaced by https://github.com/elastic/apm/issues/180