Jaeger-operator: Configure external Cassandra instance on cloud platform

[MarianZoll]

Bug description
To increase operational efficiency, we do want to host Cassandra as a storage backend outside the cluster on Azure. Cassandra on Azure does provide a host, port, username and password which should be everything that we need to create a connection for Jaeger.

Unfortunately, it seems like Jaeger does not longer take the CQLSH_PORT into consideration. Instead, the host is transferred as servers but the port is neglected. Further, we are not longer possible to require TLS on this connection. Both should have been possible for the Helm installation which is deprecated.

We did check on the storage definition to see whether we missed something in the official docs:
https://github.com/jaegertracing/jaeger-operator/blob/master/pkg/storage/cassandra_dependencies.go

Expected behavior
Guess it would be nice to have a configuration option to supply an external Cassandra service outside of the cluster. So the definition of Cassandras host, port as well as TLS options should be possible.

Steps to reproduce the bug
We tried both Operator yamls:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: istio-system
spec:
  strategy: allInOne
  storage:
    type: cassandra
    options:
      cassandra:
        servers: {{NAME}}.cassandra.cosmos.azure.com:10350 
        keyspace: "jaeger-tracing-cassandra-keyspace"
        username: {{USERNAME}}
        password: "{{PASSWORD}}"

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: istio-system
spec:
  strategy: allInOne
  storage:
    type: cassandra
    options:
      cassandra:
        servers: {{NAME}}.cassandra.cosmos.azure.com
        keyspace: "jaeger-tracing-cassandra-keyspace"
        username: {{USERNAME}}
        password: "{{PASSWORD}}"

We would really appreciate information if this is a bug or if it works as intended and we have to rely on a cluster internal storage.

Thanks in advance!

[jpkrohling]

Are you able to configure Jaeger on your local machine (bare metal or Docker) to connect to your target Cassandra cluster? If so, it should also be possible with the operator. If it's a limitation in Jaeger itself, I'll move this issue to that repository.

[MarianZoll]

So after some additional investigation I guess it is kind of both. So I am able to connect to the external cassandra using the following Helm installation on AKS:
helm install jaeger -n istio-system jaegertracing/jaeger -f helm-yamls/jaeger-values.yml

where the values are:

provisionDataStore:
  cassandra: false

storage:
  cassandra:
    host: {{NAME}}.cassandra.cosmos.azure.com
    port: 10350
    user: {{USERNAME}}
    password: "{{PASSWORD}}"
    keyspace: "jaeger-tracing-cassandra-keyspace"
    env:
      CQLSH_PORT: 10350

This spins up the required pods, but the connection is failing due to the SSL connection creation I guess:

$ kubectl logs -n istio-system jaeger-cassandra-schema-5mwdt                                                                                                                    
Connection error: ('Unable to connect to any servers', {'52.169.219.183': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})
Cassandra is still not up at ecedevweubugsvtops.cassandra.cosmos.azure.com. Waiting 1 second.

This is something that should be fixed by proper TLS certificate injection (ideally, we would be able to also configure TLS easily for certs which are from well known CAs without providing the secret at all). For the regular TLS setup with a TLS secret, there is the following discussion and PR open in the Jaeger project:
https://github.com/jaegertracing/helm-charts/issues/15
https://github.com/jaegertracing/helm-charts/pull/145

Nevertheless, the server does respond but the database connection cannot be created.

What I investigated as well is that the storage.cassandra.port is completely ignored at all by the Cassandra client. Instead only adding custom envs directly solved the issue.

So I guess we have to figure out a couple of things:

• Fix the port setup in Jaeger
• Enable easy TLS configuration
• Adopt the operator to take on the configuration

To detail the easier TLS configuration, we might want to supply a configuration, which configures the connection with a similar setup like the one described in the Python snippet. It's python because that one was handy on the Azure Quick Start tab.

    from cassandra.cluster import Cluster
    from ssl import PROTOCOL_TLSv2, CERT_REQUIRED

    ssl_opts = {
        'ca_certs': '<path-to-your-.pem-file>',
        'ssl_version': PROTOCOL_TLSv2,
        'cert_reqs': CERT_REQUIRED  # Certificates are required and validated
    }

    auth_provider = PlainTextAuthProvider(username={{USERNAME}}, password="{{PASSWORD}}")
    cluster = Cluster("{{NAME}}.cassandra.cosmos.azure.com", port = 10350, auth_provider=auth_provider, ssl_options=ssl_opts)
    session = cluster.connect("jaeger-tracing-cassandra-keyspace")

If one of you guys are pointing us to the right files we might be able to add a couple lines of code :)

[jpkrohling]

Fix the port setup in Jaeger

This is only about the create-schema job, right? If so, you'll probably need something like this to obtain the port:

https://github.com/jaegertracing/jaeger-operator/blob/6accbc2e1bd143bfea760d0c19e44529246e9981/pkg/storage/cassandra_dependencies.go#L39-L43

And then, set the env var to the job, like:

https://github.com/jaegertracing/jaeger-operator/blob/6accbc2e1bd143bfea760d0c19e44529246e9981/pkg/storage/cassandra_dependencies.go#L141-L162

Finally, you'll need to add support for the new env var here:

https://github.com/jaegertracing/jaeger/blob/043d00cc9402378c886aeace116d85e578a44834/plugin/storage/cassandra/schema/docker.sh#L6-L15

Enable easy TLS configuration

I suppose this is again only for the create-schema, as it should be relatively easy to add volumes/volume mounts with the TLS certs and use them as part of --cassandra.tls.ca. We don't seem to honor that either for the create-schema, and apparently, the create schema supports an env var named CQLSH_SSL. Do you know what this is about? Hopefully we can just use the same solution as the port above. You did mention recognizing well-known CAs by default: a good place to add this would be the same script I linked above: if the CQLSH_SSL isn't set, you could try to lookup the certs from a well-known place. We use cassandra:3.11 as the base image, not sure what the 'well-known` place is for that, but shouldn't be that hard to figure out.

[MarianZoll]

Ok we will create a PR for it. Keep you posted.

[jpkrohling]

Reopening, as the "well-known TLS" part is still pending.

[jpkrohling]

@Ashmita152, I think there are two things that could be pursued here:

1. the --cassandra.tls.ca flag should be honored by the create-schema script
2. if there are certs available in a well-known place, use them. I'm not sure what's a well-known place, though: perhaps take a look at the Cassandra images and see if it ships with certs somewhere?

Each one of those should be a different issue/PR, and I would focus on the 1st only, at the moment.

[Ashmita152]

Hi @jpkrohling

Thank you for the detailed response. I looked at the cassandra image earlier. It is based on Ubuntu image in which the standard place for ca-certificates is /etc/ssl/certs/ca-certificates.crt. I will work on it and raise two PRs accordingly.

[jpkrohling]

Wait to work on the second, we might need to have a better understanding of the whole situation there: it might not be appropriate to load a set of certificates for the create-schema and not load the same set with the other components.

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.