Jaeger: Why not use ScyllaDB?

[sunface]

It seems a perfect big data solution in APM area.

[badiib]

we're not against it, but we have been using Cassandra for quite some time now. If you'd like to provide a proof of concept or add a scyllaDB implementation, we're all for it.

[jbdalido]

Hi guys, we are launching a scylladb backed cluster right now, we just have a few issues to solved :

• TimeWindowCompactionStrategy is not yet supported by scylladb, only DateTieredCompactionStrategy for now
  https://github.com/uber/jaeger/blob/0311601ef428f9b4d29975aafc11d34a1bae2c86/cmd/collector/app/builder/span_handler_builder.go#L169-L172
  scylla is running cql in proto3 version so we have to disable this fix.

Otherwise it should be fully compatible.

[badiib]

@jbdalido right before we open sourced, we were using DateTieredCompactionStrategy, there is some performance benefit but I'd be lying if we said we exhaustively checked performance benchmarks. You should be fine. We would love to see PRs for any ScyllaDB-specific modifications.

[jbdalido]

@badiib Thanks, we will do it soon, it's in production right now, the only things we are missing is the Custom Index https://github.com/uber/jaeger/blob/master/plugin/storage/cassandra/cassandra3v001-schema.sh#L190, unavailable in ScyllaDB. if it's ok without it, i'll PR asap the removal of the proto-version and the schema specific file.

[robdefeo]

Has there been a recent update on this I see that the PR has stalled?

[isaachier]

@yurishkuro this might be faster than Cassandra. Can't vouch for compatibility, but seems worth looking into.

[prune998]

Any news ?

[isaachier]

Unless @jbdalido updates this, no.

[xmm1989218]

any updates? @jbdalido

[xmm1989218]

I found the new scylladb has been support TimeWindowCompactionStrategy, I use the newest scylladb with --experiment 1, it works well, and the performance is awesome

[xmm1989218]

sorry I found new issue "dependances" page is not support by scylladb, because of cassandra-spark-connector will use "ALLOW FILTERING", for performance reason, scylladb <= 2.3 is not supported, on scylladb offical website say it will support at 2.4.

[yurishkuro]

fyi we plan to remove that part of the schema in dependencies and make it compatible with Cassandra <3.4 (no SASI indices)

[xmm1989218]

cool

[Dudesons]

@yurishkuro Do you have an ETA when we can use scylla as a storage backend ?

[yurishkuro]

blocked in https://github.com/jaegertracing/jaeger/pull/793

[PeterCorless]

Scylla Open Source 3.0 is currently nearing release; within the month. It will support ALLOW FILTERING. (Note: there will not likely be a Scylla Open Source Release 2.4; all references to 2.4 should refer to 3.0 instead.)

[vprithvi]

SASI indices are removed by #1328

[jpkrohling]

Who wants to give it a try and report back? :-)

[dorlaor]

Scylla 3.0 with allow filtering and TWCS is there. We don't have SASI, is this a must?

[yurishkuro]

The requirement for SASI has been relaxed in 1.10, you can opt in to a data model that does not use SASI (will become the default going forward). Are there other incompatibilities?

[varunpalekar]

I have tested latest Jaeger 1.11 with scylla 3.0.4 and all things working fine.
Used following docker-compose file to deploy, you guys can also test and if things goes right then I think we don't need to change anything.

version: '3'
services:
  collector:
    image: jaegertracing/jaeger-collector:1.11
    environment: 
      SPAN_STORAGE_TYPE: cassandra
      CASSANDRA_SERVERS: cassandra
      CASSANDRA_KEYSPACE: jaeger_v1_test
    links:
      - cassandra
      - cassandra2
      - cassandra3
  agent: 
    image: jaegertracing/jaeger-agent:1.11
    command: 
      # - --collector.host-port=collector:14267
      - --reporter.grpc.host-port=collector:14250
      - --log-level=debug
    ports:
      - 5775:5775/udp
      - 6831:6831/udp
      - 6832:6832/udp
      - 5778:5778/tcp
    links:
      - collector
  web:
    image: jaegertracing/jaeger-query:1.11
    ports:
      - 16686:16686
      - 16687:16687
    environment: 
      SPAN_STORAGE_TYPE: cassandra
      CASSANDRA_SERVERS: cassandra
      CASSANDRA_KEYSPACE: jaeger_v1_test
    links:
      - cassandra
      - cassandra2
      - cassandra3


  cassandra-schema:
    image: jaegertracing/jaeger-cassandra-schema:1.11
    links:
      - cassandra
      - cassandra2
      - cassandra3
    environment: 
      CQLSH_HOST: cassandra
      DATACENTER: test
      MODE: test

  cassandra:
    image: scylladb/scylla:3.0.4
    volumes:
      - .docker/scylladb/1:/var/lib/scylla
  cassandra2:
    image: scylladb/scylla:3.0.4
    command: --seeds=cassandra
    links:
      - cassandra
    volumes:
      - .docker/scylladb/2:/var/lib/scylla
  cassandra3:
    image: scylladb/scylla:3.0.4
    command: --seeds=cassandra
    links:
      - cassandra
    volumes:
      - .docker/scylladb/3:/var/lib/scylla

[jpkrohling]

@varunpalekar if you have that as a blog post, we could certainly help spread the word about it!

[varunpalekar]

@jpkrohling I have added my code in github repo https://github.com/varunpalekar/jaeger-with-ScyllaDB
Soon also add more information in readme

[joe-elliott]

To add a datapoint: I have just setup jaeger 1.14.0 using scylla 3.1.1 as a backend without issue. We are currently ingesting ~5k spans a second but want to up that.

If I do run across any problems I will report them here.

[joe-elliott]

We continue to see success with Scylla. I will warn anyone interested in using Jaeger with Scylla of this issue: https://github.com/scylladb/scylla/issues/5168. Jaeger uses TTL expiry and a TimeWindowCompactionStrategy as reported in the issue.

It is not impacting us in 3.1.1, but it appears to impact versions at least as new as 3.0.10.

[elgalu]

@joe-elliott https://github.com/scylladb/scylla/issues/5168 is closed/fixed

[EyalDay]

We continue to see success with Scylla. I will warn anyone interested in using Jaeger with Scylla of this issue: scylladb/scylla#5168. Jaeger uses TTL expiry and a TimeWindowCompactionStrategy as reported in the issue.

It is not impacting us in 3.1.1, but it appears to impact versions at least as new as 3.0.10.
@joe-elliott Can you please share your insights regarding performance, and memory/cpu consumption when using Scylla rather than Cassandra?