Zipkin: Support Elasticsearch 5

[adriancole]

Lets work towards elasticsearch 5, from both a functionality perspective and also a classpath-dodging one. For example, our server is used as a base for others. We need to be careful about class conflicts with ES libraries, and as always guava.

Here's a note about status quo on elasticsearch-http and ES5 from @sethp-jive

Fairly well – they got rid of the string data type (I think we want keyword, since we're treating ES as a big String.equals engine and not doing "full text" search), but with a few changes I can get many of the tests to pass. The rest fail because ES goes into overdrive and pegs my CPU without returning any results within 10 seconds. The errors it does log suggest I've missed some important fields (i.e. "traceId" is still "text" and not "keyword"), but I stopped there for now.

[adriancole]

cc @openzipkin/elasticsearch ps I've started resurrecting my old okhttp work towards this. The client design that @sethp-jive made makes it much lighter effort.

I'm personally interested in an okhttp option for a number of reasons:

• We want a single library for ES http 2-5 if possible. Each dependency makes life difficult, and I'm pretty sure ES 2 will be a problem in the same classpath as ES 5.
• Similar, but focused on Jest. Library compatibility bites me in every project used as platform code and jest uses many of the usual suspects of api compat problems: guava, gson and apachehc. OkHttp (and similar like moshi) have the same dependency policy: switch package names when you break api.
• I really don't like how inefficient the apachehc stuff is, especially when using the AWS interceptor, it literally copies the data when we don't need to. While less concerning for query, we should be careful about performance on the write side.
• We want self-tracing for ES performance and behavior debugging. OkHttp already has a brave integration, as well a way to tag calls.
• And should be said.. I like to enjoy coding and OkHttp is much more fun than worrying about AWS SDK library gotchas, terrible apachehc interceptors, things like constructor order in Jest, or other indirection.

[adriancole]

I've completed the okhttp storage layer for elasticsearch 2.x (haven't looked into 5.x portability, yet). I've also written a tracing interceptor so that we can debug the performance.
https://github.com/openzipkin/brave/pull/242

After I get an okhttp aws signature v4 interceptor together, I'll raise a pull request. should be tomorrow or next day. here's a peek for now.

[adriancole]

I'm looking into this today, since ES is out

[adriancole]

currently, the biggest problem is an artificial one (the default lookback being equal to endTs).

In the current implementation, daily buckets are listed individually, so during tests this ends up looking for 15K daily indexes, which pretty much crashes the ES server.

[adriancole]

index problem is sorted. next up is how we do aggregations when using 128-bit trace IDs, considering ES5 has caveats on string fields https://github.com/openzipkin/zipkin/issues/1379

[dragontree101]

now does zipkin 1.14 version support es 5 ? i want to upgrade cassandra to es 5.