Apm: Destination service name (part 2)

[roncohen]

This issue replaces https://github.com/elastic/apm/issues/174

For the service maps feature, we need to report the logical destination of a span in order to discover the unique connections between services without processing every single span. We also need to have a field that will eventually help detect if two non-instrumented services are the same. Sameness detection will not be perfect in this iteration, but it will be good enough for the first version.

These three fields will be added to span documents. All should be keywords:

• span.destination.service.resource
  We'll use it for detecting unique destinations from each service
  For HTTP, this is the address, with the port (even when it's the default port), without any scheme. For other types of connections, it's just the span.subtype (kafka, elasticsearch etc.). For messaging, we additionally add the queue name (see examples below).
• span.destination.service.name
  We’ll use “name” for detecting “sameness” of services and then the display name of a service in the Service Map. Eventually, agents might connect to clusters and get a cluster name or similar and we could use that to detect "sameness". For now, for HTTP use scheme, host, and non-default port. For anything else, use span.subtype (postgresql, elasticsearch).
• span.destination.service.type
  For displaying icons or similar. This should be equal to the span.type

The Intake API fields will be:

• context.destination.service.resource
• context.destination.service.name
• context.destination.service.type

Another proposal used the first address in the list given on client instantiation to detect sameness of services. This is problematic because it's not always possible to retrieve those addresses. As mentioned above, eventually agents might connect to clusters and get a cluster name or similar and we could use that to detect "sameness" and put it in context.destination.service.name. Until then, we have no way to know if for example a DB service that two different services talk to are the same. The solution for now will have to be that we only show external services on the Service Map view that is "zoomed-in" on a single service - and then we don't show external services for services other than the zoomed in service.

When to set destination.service.* fields?

Generally, any spans that indicate that we're talking to another service of some kind. For local embedded databases (SQLite for example), you don't need to fill it. If you can't know (or if it's very complicated for the agent to find out) if this span is being served by a local SQLite or a remote PostgreSQL, go ahead and fill it out.

Examples

HTTP(S)

context.destination.service.resource: Capture host, and port.
context.destination.service.name: Capture the scheme, host, and non-default port.
context.destination.service.type: Same as span.type

Example 1: capture scheme, host and non-default port; omit userinfo, path, and query

Request URI: "http://user:[email protected]:1234/path?query"
Resulting context:

• context.destination.service.name: "http://testing.local:1234"
• context.destination.service.resource: "testing.local:1234" (note: no scheme)
• context.destination.service.type: "external

Example 2: omit default port in name

Request URI: "https://www.elastic.co:443/products/apm"

Resulting context:

• context.destination.service.name: "https://www.elastic.co" (default port omitted)
• context.destination.service.resource: "www.elastic.co:443"
• context.destination.service.type: "external

Note: ports should always be in address, even if the request is using the default port.

Example 3: capture IPv6 address

Request URI: "http://[::1]/"

Resulting context:

• context.destination.service.name: "http://[::1]"
• context.destination.service.resource: "[::1]:80"
• context.destination.service.type: "external

Elasticsearch, Redis, Databases, Messaging server

Example 1: Kafka cluster

queue: myqueue

Resulting context:

• context.destination.service.name: "kafka"
  use the span.subtype
• context.destination.service.resource: "kafka/myqueue"
  use the span.subtype, add / and queue name
• context.destination.service.type: "messaging

Example 2: Postgres DB

host: "server1:2424"

Resulting context:

• context.destination.service.name: "postgresql"
• context.destination.service.resource: "postgresql"
• context.destination.service.type: "db”

Future work

As mentioned, we can keep iterating on these attributes, but this should get us started.

[mikker]

... and remember to strip auth info (pass at least) from destination.service.name.

[axw]

@roncohen

apm.destination.service.type
For displaying icons or similar. This should be equal to the span.type

Do you expect this always to be the case? If so, is it useful to have the field duplicated? Can we just use span.type?

If not, do you think it would be reasonable to make this field optional in the intake API, and have the server default to using span.type? That'll save a small amount of space for most outgoing spans.

[axw]

Another, unrelated thing (in case it's not obvious, I'm implementing this now ;))

For database spans, I think we should not set destination.service.* when we can tell we're dealing with an embedded database, like SQLite.

[roncohen]

Good input @axw.

Do you expect this always to be the case? If so, is it useful to have the field duplicated? Can we just use span.type?
If not, do you think it would be reasonable to make this field optional in the intake API, and have the server default to using span.type? That'll save a small amount of space for most outgoing spans.

At the moment, that's the case. Down the line, maybe we'll let users override it for certain situations, but i agree it's a bit far out. Never the less, let's stick to explicit fields.

I'd rather not introduce more if-not-set-default logic in APM Server. It becomes hard to reason about and in the grand scheme of things, we'd only be saving a relatively small bandwidth amount between agent and server and a bit of serialization/deserialization.

For database spans, I think we should not set destination.service.* when we can tell we're dealing with an embedded database, like SQLite.

I can already hear the questions like: "What do to when we cannot know if it's a SQLite DB or not". I'm OK to say that if you _know for sure_ it's not an external service, don't fill the destination.service.*, but if you're in doubt, go ahead and fill it out. I'll update the description.

[vigneshshanmugam]

For the RUM agent, we create two types of spans for external calls resource (which are static resources) and external (Fetch and XHR calls), Will it be useful to set the JS, CSS and Images spans types to external on the destination.service.type field or keep it as resource similar to span.type ?

I was worried if it would confuse users while viewing the service maps if we have it as resource. @roncohen Thoughts? https://github.com/elastic/apm-agent-rum-js/pull/515#discussion_r355929842

[axw]

More questions (sorry):

I started looking at adding the fields to the spec and ES schema (https://github.com/elastic/apm-server/issues/3025). A couple of things felt a little odd to me:

• Nesting under apm. in the ES schema, rather than under span. like other span context.
• Nesting under span.context.destination in the intake, rather than just span.context.service, like how we have span.context.http and span.context.db. Isn't destination implied?

EDIT: span.context.service already exists as a means of overriding service info from payload metadata, so it's not possible to change that. I just propose to change apm. to span. then.

I'm wondering if we can amend the field names like so (sorry sorry): Intake: - `context.service.type` - `context.service.name` - `context.service.resource` Elasticsearch: - `span.service.type` - `span.service.name` - `span.service.resource` https://github.com/elastic/apm/issues/115 would remain unchanged.

[roncohen]

I’m ok with that. Will you update the description?

[roncohen]

@vigneshshanmugam

For the RUM agent, we create two types of spans for external calls resource (which are static resources) and external (Fetch and XHR calls), Will it be useful to set the JS, CSS and Images spans types to external on the destination.service.type field or keep it as resource similar to span.type ?
I was worried if it would confuse users while viewing the service maps if we have it as resource. @roncohen Thoughts? elastic/apm-agent-rum-js#515 (comment)

Good question. I think it'll be use to keep it separate. The user will be curious to know which hosts we pull "dumb" resources from and which hosts they potentially have more rich interactions with?

It brings up another question around what to do if you have resources served from the same domain as the main page. E.g. a request to a resource at a relative location. In a sense, we should show it because it might slow down your site. On the other hand, you can also imagine serving the same page from many different domains. If you have resources (or external for that matter) serviced locally, each domain will show up separately. We could consider setting <local> or similar for any resources served from a relative URL. Thought?

[vigneshshanmugam]

Good question. I think it'll be use to keep it separate. The user will be curious to know which hosts we pull "dumb" resources from and which hosts they potentially have more rich interactions with?

Agree with you on this, since the user can check if third party resources are slowing down their site and should be also be filter out dumb resources.

If you have resources (or external for that matter) serviced locally, each domain will show up separately. We could consider setting or similar for any resources served from a relative URL. Thought?

Currently, the span names for the requests with relative urls would have GET /api/data and others would have the full URL path GET www.example.com/api/data. Wouldn't that already provide a distinction. Should we need to consider another setting?

[eyalkoren]

@roncohen in RDBMS it is common to have a _database_ property, which is a namespace containing db objects (eg tables, indices), so you can have a table named "foo" in database "db1" and "db2".

In some cases there is also the notion of a DB _instance_, that represents the db runtime (memory and processes required to serve db requests).

I think it makes sense to append the _database_ or _instance_ to span.destination.service.resource if we discover such (eg postgresql/mydatabase; mysql/test_instance), WDYT?

Also, regarding this:

For local embedded databases (SQLite for example), you don't need to fill it. If you can't know (or if it's very complicated for the agent to find out) if this span is being served by a local SQLite or a remote PostgreSQL, go ahead and fill it out.

Do you intend to filter out destinations when building service maps if span.context.destination.address's value is localhost or 127.0.0.1 or ::1? It may take care of a lot of these cases.
Maybe it even makes sense to filter out DB spans that do not contain destination.address. If we fail to discover the DB address from the connection string, this may be a good indication that it is used in-memory/locally.

[roncohen]

I think it makes sense to append the database or instance to span.destination.service.resource if we discover such (eg postgresql/mydatabase; mysql/test_instance), WDYT?

Currently, the only use we have for span.destination.service.resource is to help differentiate connections when building the service map. I want to make sure that we're only adding things we have a direct use for at this point. Are you thinking people will be using a database as the backing for a task system and that's why you want to add it?

[eyalkoren]

I think it is very likely that when users are using multiple database servers/clusters in their multi-service environment, they will have different instances and/or database namespace.
Currently, if an environment is using multiple MySQL databases for different purposes, our filter of only mysql can easily miss some of those, which are really different connections.
Anyway, it is not easy to find in all cases, so it would be optional in any case. I can't see a reason not to do that, as it is merely a string you use for filtering in your searches, right?

[roncohen]

@eyalkoren Sorry, I probably wasn't clear. We are using span.destination.service.resource to discover _connections_ between _instrumented services_. We are not going to use it to discover external services themselves at this point. For that reason, we should not spend time on adding database name to span.destination.service.resource now. Let's discuss this again when we have the first beta version out

[eyalkoren]

OK, I think you said that before, but seeing it as part of this issue made me think we are going to use this info for that.

[basepi]

422 covers the spec creation for destination service info. I'm going to close this issue in favor of that one.