Jaeger: Do not force cassandra protocol's compressor

[sylr]

When using cassandra, Jaeger force the use of the SnappyCompressor for the cassandra protocol.

https://github.com/jaegertracing/jaeger/blob/4ada2b924bfcd393ff370506b9bab8ced1c72376/pkg/cassandra/config/config.go#L127

Azure Cosmos DB provides a cassandra compatible API but it does not seem to be compatible with SnappyCompressor (see https://github.com/gocql/gocql/issues/1205).

[yurishkuro]

I don't remember, what is the compressor even used for? Compressing on the wire or at rest?

[sylr]

@yurishkuro given that the Compressor is defined in a connection struct I would say it's for compressing on the wire:

https://github.com/gocql/gocql/blob/fb832297d742a757f1e631aded19c373e35dc67d/conn.go#L94-L103

[black-adder]

Would you be open to creating a PR that makes this configurable?

[sylr]

FYI Azure Cosmos DB does not allow "CREATE TYPE" so it won't work anyway.

[486]

@sylr yes Cosmos DB does support CREATE TYPE, we tried it. It is just not listed.
So I think making it configurable is still a good idea.

[xihw]

Hi @sylr / @black-adder I also confirmed Cosmos DB does support CREATE TYPE now. Would be of great help if you could make it configurable.

[sylr]

Sorry guys, we switched to elastic for the data storage and I no longer have time to test CosmosDB.

[486]

@black-adder @yurishkuro we hacked on it and made it work with CosmosDB... mostly.

The point is that CosmosDB in Cassandra mode has some specialties that we need to work around. In the end it would become a distinct backend type rather than a few config switches.

Before we put more work into it, would you be open to accept a new backend?

It would surely make jaeger very convenient to use on Azure.

[xihw]

@486 good to hear that you are able to make it work. Would you mind sharing more information on what work around you did ?

For me it always says:
{"level":"info","ts":1541018500.2958531,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":16687,"status":"unavailable"} {"level":"fatal","ts":1541018505.7411027,"caller":"query/main.go:105","msg":"Failed to init storage factory","error":"gocql: unable to create session: unable to fetch peer host info: EOF","stacktrace":"main.main.func1\n\t/Users/jhon/go/src/github.com/jaegertracing/jaeger/cmd/query/main.go:105\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/Users/jhon/go/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:698\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/jhon/go/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:783\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/Users/jhon/go/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:736\nmain.main\n\t/Users/jhon/go/src/github.com/jaegertracing/jaeger/cmd/query/main.go:172\nruntime.main\n\t/usr/local/Cellar/go/1.11.1/libexec/src/runtime/proc.go:201"}

But I definitely can connect to it with cqlsh.

[yurishkuro]

@486 new backend is feasible, but we prefer not to have them in the main repo, rather as a plugin (depending on #422).

[lukasmrtvy]

Seems its working:

docker run -d --restart always \
--name jaeger-query \
-e CASSANDRA_SERVERS=myserver.cassandra.cosmosdb.azure.com \
-e CASSANDRA_PORT=10350 \
-e CASSANDRA_PASSWORD=mypass \
-e CASSANDRA_USERNAME=myaccount  \
-e CASSANDRA_TLS=true \
-e CASSANDRA_KEYSPACE=jaeger_v1_dc1 \
-e CASSANDRA_CONNECTIONS_PER_HOST=1 \
-e CASSANDRA_TLS_VERIFY_HOST=false \
jaegertracing/jaeger-query:latest

Problem is with populating schema:

docker run --rm -it -e CASSANDRA_USER=myuser -e CASSANDRA_PASS=mypass -e CASSANDRA_HOST=myserver.cassandra.cosmosdb.azure.com -e SSL_VALIDATE=false -e SSL_VERSION=TLSv1_2 -e keyspace=jaeger_v1_dc1 -e replication=1 -e trace_ttl=172800 -e dependencies_ttl=0 --entrypoint bash jaegertracing/jaeger-cassandra-schema:latest -c 'sed  -e "s/--.*$//g" -e "/^\s*$/d" -e "s/\${keyspace}/${keyspace}/" -e "s/\${replication}/${replication}/" -e "s/\${trace_ttl}/${trace_ttl}/" -e "s/\${dependencies_ttl}/${dependencies_ttl}/" /cassandra-schema/v001.cql.tmpl > /cassandra-schema/v001.cql ; cqlsh ${CASSANDRA_HOST} 10350 -u ${CASSANDRA_USER} -p ${CASSANDRA_PASS} --ssl -f /cassandra-schema/v001.cql'

/cassandra-schema/v001.cql:11:SyntaxException: line 9:240 no viable alternative at input ') (...ring    text,
    value_bool      boolean,
    value_long      bigint,
    value_double    double,
    value_binary    blob,
)...)
/cassandra-schema/v001.cql:15:SyntaxException: line 4:102 no viable alternative at input ') (...    ts      bigint,
    fields  list<frozen<keyvalue>>,
)...)
/cassandra-schema/v001.cql:20:SyntaxException: line 5:131 no viable alternative at input ') (...      text,
    trace_id        blob,
    span_id         bigint,
)...)
/cassandra-schema/v001.cql:24:SyntaxException: line 4:120 no viable alternative at input ') (...service_name    text,
    tags            list<frozen<keyvalue>>,
)...)
/cassandra-schema/v001.cql:48:SyntaxException: line 9:243 no viable alternative at input 'duration (...gint,
    parent_id       bigint,
    operation_name  text,
    flags           int,
    start_time      bigint,
    duration...)
/cassandra-schema/v001.cql:61:InvalidRequest: Error from server: code=2200 [Invalid query] message="gc_grace_seconds value must be zero."
/cassandra-schema/v001.cql:75:InvalidRequest: Error from server: code=2200 [Invalid query] message="gc_grace_seconds value must be zero."
/cassandra-schema/v001.cql:91:InvalidRequest: Error from server: code=2200 [Invalid query] message="gc_grace_seconds value must be zero."
/cassandra-schema/v001.cql:107:InvalidRequest: Error from server: code=2200 [Invalid query] message="gc_grace_seconds value must be zero."
/cassandra-schema/v001.cql:125:SyntaxException: line 5:299 no viable alternative at input 'duration (...  text,      // service name
    operation_name  text,      // operation name, or blank for queries without span name
    bucket          timestamp, // time bucket, - the start_time of the given span rounded to an hour
    duration...)
/cassandra-schema/v001.cql:144:InvalidRequest: Error from server: code=2200 [Invalid query] message="gc_grace_seconds value must be zero."
/cassandra-schema/v001.cql:149:SyntaxException: line 5:133 no viable alternative at input ') (...      text,
    child           text,
    call_count      bigint,
)...)
/cassandra-schema/v001.cql:161:InvalidRequest: Error from server: code=2200 [Invalid query] message="Unknown type jaeger_v1_dc1.dependency"
/cassandra-schema/v001.cql:164:ServerError: 'CREATE CUSTOM INDEX' is not supported yet.

EDIT:
These commas causing problems https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/v001.cql.tmpl#L32
CosmosDB does not support Indexes yet:
https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/cassandra/schema/v001.cql.tmpl#L205
https://docs.microsoft.com/en-us/azure/cosmos-db/cassandra-support#cql-commands

Also gc_grace_seconds

Note gc_grace_seconds value must be zero if specified.

[kanshiG]

Custom index except SASI are supported from this week onwards for all accounts.

[yurishkuro]

Jaeger doesn't use any Cassandra indices