Jaeger-operator: Expose collector/agent outside of the cluster

[atmosx]

Hello! I'd like to ask if collector service type and annotations are accessible through the config, e.g.:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  collector:
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        external-dns.alpha.kubernetes.io/hostname: "my.hostname.com"
      type: LoadBalancer

The above config doesn't work, I don't see annotations or type to collector service.

Ty!

[objectiser]

@atmosx No these are not currently defined in the CRD. Annotations can be defined at the top level, or under the collector node, but these are associated with the collector's Deployment object.

So we would need to consider how best to distinguish annotations where two or more possible objects may be relevant - i.e. in terms of the collector it could be for the service or deployment objects.

May be the best approach is as you have illustrated above - creating a new service node under the binaries.

@jpkrohling @pavolloffay Any thoughts?

[pavolloffay]

hi @atmosx could you describe your use-case? Why do you want to use an external load balancer?

[atmosx]

@pavolloffay Hello! There is a set of services that run on the same AWS VPC as the kubernetes cluster, but outside the kubernetes cluster. These services have to send traces to the collector, which runs in the kubernetes cluster.

[pavolloffay]

Did you try using ingeress? Jaege operator should support that.

[atmosx]

I tried using ingress for the query service, but since it doesn't support hosts did not play well with traefik, I disabled the ingress and used an ingress definition instead.

How can I use ingresses to expose collector ports? Can you show me an example config so I can try myself?

[pavolloffay]

I think we have not covered yet exposing collector outside of the cluster. I found only this recommendation for OCP https://github.com/jaegertracing/jaeger-openshift#exposing-jaeger-collector-outside-of-cluster

Maybe as a workaround you could create a different service with LoadBalancer type.

[objectiser]

@jpkrohling A topic for discussion when you are back in the office :smile:

[atmosx]

Maybe as a workaround you could create a different service with LoadBalancer type.

That's what I did and seems to work! Thanks for the replies!

[jpkrohling]

I'm not sure it makes sense to expose the agent outside of the cluster, but I can certainly see the case for the collector. I wouldn't do that before we have clear instructions on how to protect the collector, though.

[atmosx]

Hello! There is a number of different ways to protect the collector on a private network:

• network L3/4 (firewalls, AWS security groups, etc)
• SSL certificates network L5

I don’t think that security in scope though.

What is the worst thing a rogue agent can do? DoS the collector and/or pollute the logs?

[jpkrohling]

DoS the collector and/or pollute the logs?

In general, whatever is exposed to the public internet should be secured. When the collector is exposed, your interface isn't only with agents, but also with malicious clients.

The collector has write permissions to the storage, as well as access to the internal cluster network. An unsecured collector could be used as an attack vector, especially in combination with other (unknown) vulnerabilities.

DoS would be one valid concern as well, as one could just write garbage until the storage is full.

And finally, you would be surprised to see what spammers are up to these days :-) As someone debugging a production issue on a real service, the last thing I would want is to find a buy-bitcoins service.

[atmosx]

Thanks for the answer.

Our k8s cluster lives within an AWS VPC. Exposing an unprotected service to the internet never crossed my mind in all honesty. The idea was to expose the collector to services running within the VPC. We use Security Group IDs to limit connectivity only to the hosts that _should_ have access to the collector.

Your point about security is valid though. It is not clear to me what do you mean by:

I wouldn't do that before we have clear instructions on how to protect the collector, though

are you talking about integrating authN/Z backends to the collector or adding documentation about possible ~drawbacks~ dangers of security negligence along with solutions/best practices?

[jpkrohling]

It is not clear to me what do you mean by:

I wouldn't do that before we have clear instructions on how to protect the collector, though

During the last bi-weekly call, we talked about the need to provide instructions on how to protect the communication between agents and collectors. It should be possible today already, we just need to document it (hopefully).

My badly formulated point was that I wouldn't leave the collector open to the public internet, as we should have clear instructions in the near future on how to protect it.

[pavolloffay]

I came across this. To expose collector port as OpenShift route run this:

oc create route edge --service=<jaeger-instance-name> --port c-binary-trft --insecure-policy=Allow

[jpkrohling]

I'm closing this, as I don't think we want that, at least not for now. We haven't received any further requests about this either, and there are valid workarounds for the cases where this feature makes sense, as @pavolloffay mentioned above.

If we receive further requests for this feature, we can reconsider it.