Zipkin: Decide what to do about remote service name and search

[adriancole]

Right now, we return all service names, local and remote in /api/v2/services (and v1 for that matter). This is used for the query search drop-down, and they also correspond to the dependency graph.

We currently don't have any tests for being able to search by remote service name, so storage implementations handle this differently. I'm of the opinion we should either remove the remote service names from the list, or decide how to handle a query like this. For example, do we want to allow a query like "where I called service X" or do we want to leave like many are doing which is "either I am this service or this service was called"?

Until we do something, we should ack that we write extra data and I'm pretty sure more than one implementation only consider the local service name in queries!

cc @openzipkin/core

[xqliang]

Some times i need to known "whether calling X remote service is slowing".

[adriancole]

right, so the span name is attached to a local service (the services query
mixes local and remote service when choosing a span name). I wonder if we
shouldn't have a separate drop-down for remoteService? Or maybe the mix is
ok.. what do you think?

[xqliang]

mix is ok for me

[goller]

I like the mix option because from a UI point-of-view one is looking for a specific service regardless if it is local or remote.

I think it is better that there is a UI that allows one to "move up" towards parent spans or "down" to children.

[adriancole]

>

I think it is better that there is a UI that allows one to "move up"
towards parent spans or "down" to children.

mind raising a separate issue? maybe with a screen shot with apple preview
annotation sketch (or your favorite technique) to describe this?

[adriancole]

This issue came up again in #2337 I think that end users are paying a lot for this, and we should simply stop writing indexes for remote and local service names against the same query conditions.

One of the more damaging things conceptually is when one conflates local and remote service names, so what we do in the services index actually contributes to this.

If we do stop indexing the spans against a remote service name, we will lose the ability to search for remote services.. unless we introduce another query which is remoteService=X. I'm personally still better with this as we reduce expense and in some cases restore usability meanwhile.

[adriancole]

re-pinging @openzipkin/core @openzipkin/ui @openzipkin/cassandra as IMHO we can't keep tripping over the same problem year after year.

If you are in support of refining the service/span query to local only, please thumbs up this issue. If you don't want this change, thumbs down. If you are ok with this with a caveat such as an additional query option, please add a comment instead of down-thumbing. In any case, if you thumbs down, please make a comment why.

[llinder]

If we refine this to local only then I would vote for introducing another query. Reasoning is that we often call services we don't own and being able to search on these remote services could be helpful in some cases.

[adriancole]

>

If we refine this to local only then I would vote for introducing another
query. Reasoning is that we often call services we don't own and being able
to search on these remote services could be helpful in some cases.

Thanks @llinder. To dig into this, can you give an example query that you
would use? I'm assuming you'd use remoteServiceName the same way as if it
were a tag. A realistic query based on your test data would be fantastic

>

[adriancole]

so if we did the split thing I think it would help reduce the cardinality of @openzipkin/cassandra data considerably. What would happen is that it would be like this:

Now, we have:

localServiceName + "" (for /services endpoint)
localServiceName + spanName (for /spans?serviceName=X endpoint)
remoteServiceName + "" (for new /services endpoint)
remoteServiceName + spanName (for /spans?serviceName=Y endpoint)

The proposal would have less cardinality as we wouldn't index remoteServiceName + spanName:
localServiceName + "" (for /services endpoint)
localServiceName + spanName (for /spans?serviceName=X endpoint)
remoteServiceName + "" (for new /remoteServices endpoint)

The result would be..
/services -> only local serviceNames
/remoteServices -> new: only remote serviceNames for auto-complete
/spans?remoteServiceName=X -> new: to restore functionality @llinder mentioned

I think the impl would be somewhat easy and a good idea to do now vs later as we are already adding new autocomplete endpoints for tags. cc @openzipkin/ui

[adriancole]

ps thinking about this it might be better to make a /remoteServices?serviceName=X endpoint

that way search is dependent on the local endpoint that called the remote one. otherwise it would apply to any span in the trace (any of the perhaps hundreds of services involved).

it really depends on the intent of the query

[adriancole]

PS this particular line of change will not help with instrumentation which accidentally create high cardinality span names. For example, it might not be obvious that client and server documents are sent separately. each document includes a span name and would be indexed regardless of this change. If someone is seeing high cardinality data in span names another solution is needed. Likely deleting the span name that is high cardinality as they should never be. (ex this is why we default client span names to the http method).

[adriancole]

I have started on this and will raise a pull request soon

[adriancole]

@openzipkin/ui so I am working on this now, and it should be done not too long.

/services -> only local serviceNames
/remoteServices?serviceName=X -> new: only remote serviceNames for auto-complete
/spans?remoteServiceName=X -> new: to restore functionality @llinder mentioned

[adriancole]

one thing important here is how to handle trace query where remoteServiceName is present and serviceName is not.

Right now, the span name list is dependent on service name. However the trace query against span name works even if there is no service name indicated. This has cost to it. I plan to follow the same model (allowing queries by remote service even lacking a local service name) unless someone mentions otherwise. Personally, I can see it both ways..

On one hand, the UI will never access this index because the field values are auto-complete and they are only auto-completed when a service name is present. It seems not great to optimize and incur cost on a type of query the UI won't make

On the other hand, we do support this same thing with span names and those using api directly may have a need.

Personally, I don't like supporting it because of the cost and additional indexing complexity. However, path of least surprise seems to allow "remote service even lacking a local service name", which is why I'd proceed that route unless someone says no.

[adriancole]

https://github.com/openzipkin/zipkin/pull/2499 will complete this change

[adriancole]

I earlier made a comment that a conservative approach (making remoteService dependent on service) would allow things to progress until we had demand for the latter https://github.com/openzipkin/zipkin/issues/1794#issuecomment-481482792

Recently, #2999 was opened to allow search solely by remote service name. If you desire this, upvote it