Zipkin: Proposal: drop Scala backend implementation in favor of zipkin-java

[abesto]

The zipkin-java project was created out of issues #451 #463. It is now approaching version 1.0, and resulted in questions about the future of the scala implementation. This issue will enumerate opportunities, concerns and any action to take on this topic.

Pro

• zipkin-java has more features
  
  
  
  • It has a no-dependency library, for example in instrumentation projects, custom collectors or spark
  
  
  • fine-grained logging of dependency linking and collection stages
  
  
  • fine-grained, and tested health check system
  
  
  • collector-metrics broken out by transport
  
  
  • integration tests that invoke the packaged exec jar
  
  
  • elasticsearch storage
  
  
• Maintainers can do more, if only supporting one server framework.
  
  
  
  • Currently, there's a lot of distracting work, porting code and tests, and releasing both projects
  
  
  • Docker and documentation is confusing when there is more than one authoritative process.
  
  
  • By cutting scala, zipkin's few maintainers we can progress more with the same time, and cause less confusion.
  
  
• The java build is more stable due to less dependencies
  
  
  
  • The scala project uses more dependencies including the unreliable maven.twttr.com repository which often breaks the build with http 503 errors.
  
  
• Java implementation is made from better understood technology
  
  
  
  • the server is based on spring-boot, which is a popular framework and well understood from a configuration POV.
  
  
  • the scala process is hosted on a relatively rare framework, and literally loads .scala files to configure the process!
  
  
• Java implementation can be deployed as a single process
  
  
  
  • while not required, you can enable all features including kafka in the same process
  
  

• Java implementation has more people participating in difficult topics

  • Topics including modularity, dependency management and configuration have had fast progress in the java project.
  • elasticsearch was written in <1 week and worked both in the stock server, and one based on Armeria a netty rpc service. This is in contrast to years of requests in scala with no action.
    

    Con

• No precanned profiles like "collector"

  • Some may desire a profile like collector includes the ability to receive but not query spans.
• Scala implementation battle-hardened at Twitter. Performance, availability implications.
  
  
  
  • A lot of zipkin code isn't used at twitter (Ex cassandra, kafka), but Finagle + Scribe is.
  
  
  • Those already using zipkin collectors may have higher confidence in Finagle + Scribe vs FB swift (used in zipkin-java)
  
  

[klingerf]

Ok, I will take the pro-scribe-anti-java side of this debate :stuck_out_tongue_winking_eye:

Our primary use case for Zipkin is exporting tracing data from Finagle-based services. Finagle includes a finagle-zipkin library that implements a Zipkin tracer. Unfortunately, that tracer is only configured to work with scribe. I'm also not a big fan of scribe, and at present we bypass scribe altogether by configuring Finagle's ZipkinTracer to send data directly to a zipkin-collector process on port 9410. All of that said, we still require scribe protocol support for this setup to work.

There's an open issue with the Finagle folks tracking implementation of an http-based tracer instead: https://github.com/twitter/finagle/issues/465. That seems promising, and given the widespread use of Zipkin+Finagle, I think that upgrading Finagle is a logical precursor to removing scribe support. It's true that folks could patch their finagle-zipkin tracers to speak http, but that feels like more of a workaround than an actual fix.

Swapping out Finagle's scribe implementation with an http one brings up a larger issue that I think is worth pointing out. How do we know that an http-backed Zipkin tracer is as performant as Finagle's scribe-backed Zipkin tracer? The existing scribe tracer is widely deployed within Twitter's own infrastructure. It has been tuned and tested in thousands of super-high-traffic applications over many years. Do we have any side-by-side comparisons of the proposed http tracer versus the existing tracer?

I have a similar concern with the zipkin-java project, although I have to admit that I'm not nearly as familiar with that project, so my concerns may be unfounded. The scala-based zipkin-collector and zipkin-query processes have been heavily tuned to withstand a high volume of requests. They operate independently, such that a degradation in one process does not affect the ability of the other to serve requests. Do we have the same guarantees with an all-in-one zipkin-java process? "Lower complexity" may not actually be an improvement in this case if it has a negative impact on resiliency.

Thanks for opening this issue as a place to track discussion. Those are just my initial thoughts / questions, and I'm certainly happy to discuss in more detail.

[abesto]

@klingerf Great stuff. I tried to capture short versions in the description, feel free to edit it to clarify / correct whatever I messed up.

[klingerf]

@abesto Looks great -- thanks for summing it up.

[jtrobec]

I'd like to +1 Kevin's concerns...from Twitter's perspective, moving the fundamental infra away from scribe and scala isn't inherently problematic as much as it is labor intensive. We would really need to make sure that it scales enough to support our infra before we could make a switch.

[eirslett]

Java implementation has lower complexity in terms of technology used (TBD: how exactly?)

Java is a more widely known programming language than Scala, and the Spring framework/stack is more widely known than the Finagle/Twitter framework/stack. The technical complexity is about the same (I think), but more people know Java than Scala.

No built-in Scribe collector support, breaking feature parity

I think we should add scribe support to zipkin-java. The java port should be a drop-in replacement for the scala implementation, so we don't need to make any changes to the instrumented applications.

Multi-process Zipkin provides extra resiliency

,

They [collector/query] operate independently, such that a degradation in one process does not affect the ability of the other to serve requests.

This could be solved in the infrastructure, by only sending query traffic to some instances, and only sending collector traffic to other instances. I'm not concerned about query workloads taking down a collector (it's very low volume), but a collector which is receiving too many spans could possibly crash, and then the query functionality wouldn't work either. So for extra resiliency, you could deploy one query-only instance that is guaranteed to answer, even when the collectors are flooded. (Unless the storage backend is also kneeling)

How do we know that an http-backed Zipkin tracer is as performant as Finagle's scribe-backed Zipkin tracer?

We should write a couple of load tests:

finagle -> scribe -> zipkin-scala
finagle -> http -> zipkin-scala (if this is supported?)
finagle -> scribe -> zipkin-java
finagle -> http -> zipkin-java

and see how many spans per second we can process.
Also, measure how latency in the storage backend affects server performance, and how much traffic the collector can take before it crashes.

moving the fundamental infra away from [scribe and] scala isn't inherently problematic as much as it is labor intensive.

Given that Zipkin is distributed as a shaded jar, Scala is already an implementation detail from the infrastructure point of view; it's just a jvm process, or even just a Docker container. But given that Twitter builds Zipkin from source, for Manhattan support (is that still true?) then there would be a rewrite effort, since the APIs would be Java, not Scala.

[adriancole]

These are valid concerns, though I'd like to mention a few things that all
might not know. They impact the perception of supportability:

• zipkin's scala code has had numerous significant bugs, yet very few have
  contributed fixes to it.
• very little of the scala code is actually used inside Twitter. For
  example, twitter doesn't use cassandra.
• there is little evidence of heavy tuning in the source history of the
  scala code
• benchmarks in scala that have happened have been ad-hoc. For example,
  pyramid-zipkin adjusting parameters for kafka (these parameters exist in
  both java and scala projects)

The scala+finagle aspect has actually led directly to people not
participating in the project, and also delayed features for extreme amounts
of time. For example, elasticsearch was requested years ago, and had some
false-starts. Elasticsearch was implemented in java in less than a week by
someone who formerly had no experience with zipkin. There's a lot of
evidence we can dig up about this, and this was a primary motivation for
the java project itself.

While both the scala and java processes can enable a feature like kafka,
there's no reason to believe that either choice means a resilience fail.
Before, we had no choice for an all-in-one process: that significantly
damaged the ability for people to learn zipkin quickly. IOTW, let's not
mistake choice for a topology mandate. It is already the case that folks
have collectors in golang and whatnot. There's nothing preventing someone
from making a zipkin server that is only a collector. in fact, there
already is one in spring-cloud-sleuth for rabbit.

zipkin-java had a scribe transport, but it was deleted because scribe is
archived and folks involved in zipkin had bad experience with scribe. We can
choose to re-enable that feature in java using facebook swift (history
lesson: facebook made scribe and thrift, so this choice shouldn't be scary)

Regardless of whether folks want a scala+finagle thing or not, it is
important that zipkin lives. Right now, I spend a great deal of time on
undifferentiated work in scala eventhough I get lots of help in java. There
is a burden to maintaining zipkin and with the amount of people we have, it
surely suffers from having to carry excess weight in relatively unknown
frameworks.

[adriancole]

updated the description a bit as I noticed people were reacting to the idea that java had to be single-process when there's not actually anything about it that constrains this.

[eirslett]

We could keep the zipkin-scala module, but deprecate/sunset it - only accept bug fixes that are contributed by people who still use the Scala implementation. At the very least, let project maintainers add features in zipkin-java and not expect them to reimplement it in zipkin-scala.

They would still get any new UI updates, since the UI is just a single-page webapp that can be hosted in nginx. The exception would be if the UI relies on new API features that aren't implemented yet - maybe the /health endpoint on the server chould expose a "features" array? "features": ["dependency", "search"] and then we could adjust the UI accordingly.

[mosesn]

It sounds to me like the main concern people have is that the migration story isn't clear for everybody. I think @eirslett's idea of maintaining the zipkin-scala module, at least for now, is a good idea. The other way forward would be to get zipkin-java compatible with finagle, check that it meets zipkin-scala performance, and provide a migration guide before killing zipkin-scala. Although @adriancole points out that the scala-zipkin backend isn't the most resource-intensive process, work has gone into it to make sure that unexpected increases in span write volume doesn't overload our storage backend, which doesn't seem to be there yet in zipkin-java.

I don't think anyone is ideologically opposed to zipkin-java and it looks solid. I don't think we should block this on zipkin-java achieving feature parity with zipkin-scala. But I think it would be nice to see a clearer migration story before we drop zipkin-scala.

[eirslett]

Another question, in terms of workload, is whether we would spend more resources on deprecating zipkin-scala (it's an effort in itself) than it would take to simply maintain the two side-by-side.

[adriancole]

work has gone into it to make sure that unexpected increases in span write
volume doesn't overload our storage backend, which doesn't seem to be there
yet in zipkin-java.

I think you are referring to the adaptive sampler, if I'm not mistaken. It
sat broken for a while because it had almost no test coverage, and no users
outside twitter complaining. I think that feature should be spun out, as it
has a lot of code that hasn't been touched in years.
https://github.com/openzipkin/zipkin/tree/master/zipkin-sampler

[klingerf]

Good feedback on this thread. Thanks @adriancole for some of the backstory that I was missing in my original post.

If it's at all possible to keep scribe support around in the zipkin-java world, I think that would make life easier for a lot of zipkin + finagle customers. Facebook swift sounds like a good way to go about that. I also like the suggestion of gathering more head-to-head performance numbers by which to evaluate the viability of zipkin-java in a large-scale system. That, plus a migration guide, plus a list of adopters, would help to make the transition much smoother for folks.

[adriancole]

thanks for the interest, Kevin. For starters, zipkin-java's docker image
https://github.com/openzipkin/docker-zipkin-java was written with
transition in mind: except for scribe (which we can chase up), it responds
to all environment variables from upstream. This means that if using
docker, one can do performance comparisons, etc with http, kafka, mysql,
cassandra, etc by swapping the images.

One of the reasons for wanting to drop the scala project is that we are
swamped with maintenance. That hints that things like rigorous performance
comparisons aren't likely to happen without volunteers, particularly as
there's no benchmark of any sort in the scala project. Anyway, I started this
issue https://github.com/openzipkin/zipkin-java/issues/45 on the
generalities of things that might be helpful.

[adriancole]

added a blocking issue to add the scribe transport to zipkin-java. https://github.com/openzipkin/zipkin-java/issues/150

[adriancole]

broke out a topic on zipkin-sampler https://github.com/openzipkin/zipkin/issues/1078

[klingerf]

Cool, will watch openzipkin/zipkin-java#150 for progress and try it out when it's ready.

[adriancole]

I've resurrected scribe in the java project. once https://github.com/openzipkin/zipkin-java/pull/152 is released, I'll scrub it from the pro/con list in the description

[adriancole]

fyi zipkin-java 0.12.1 has working scribe and available in docker. cc @klingerf @jtrobec @gena01

[adriancole]

ps once we drop the scala server, we can avoid the routine build breaks due to twitter dependencies like this.

GET 'https://maven.twttr.com/com/twitter/common/stats-util/0.0.58/stats-util-0.0.58.jar'. Received status code 503 from server: Service Temporarily Unavailable

[adriancole]

zipkin-java 0.13.0 has both scribe and sampler-zookeeper (the adaptive one). there's still probably a week or two's work before 1.0 is realistic.

[adriancole]

Just rewrote the adaptive sampler to remove the scala dependency. https://github.com/openzipkin/zipkin-java/pull/167

[adriancole]

zipkin-java is headed towards the finish line, we've had no unresolved objections, and a dozen votes in favor.

I think we should focus on logistics at this point, for example explicitly suggest new projects and those with flexibility to use zipkin-java, mention more formally the bug-fix nature of this project, make explicit logistics of non-scala assets such as the thrift files and javascript. Mention the current plan which is to replace master with the assets that are currently in the zipkin-java repo. Setup an answer for the question of who's already using it, etc.

To do all this, we could either re-title/ repurpose this, or close it and make a new logistics issue. Any other ideas?

PS here's latest release notes from the java project: https://github.com/openzipkin/zipkin-java/releases/tag/0.20.0

[abesto]

Agree. I'm for closing this and creating new issue(s) to track the logistics work.

[adriancole]

closed via #1119