Pomerium: Using Pomerium as Kubernetes ingress controller

[fiahil]

Note, this has been discussed here #94

Is your feature request related to a problem? Please describe.

I'm using Pomerium inside a Kubernetes cluster with nginx-ingress as the ingress-controller. Currently, requests are proxied twice (first Nginx, and then Pomerium proxy) and pomerium policies look a lot like ingress definitions.

Describe the solution you'd like

Ideally, we would like to choose between two solutions:

• Use Pomerium as external authentication/authorization service for an existing ingress-controller (ingress-nginx, traefik, ambassador)
• Use Pomerium as ingress controller by leveraging existing proxies or load balancers like Envoy

Describe alternatives you've considered

I've been testing multiple ingress controllers, and the most suitable alternatives would be Kong or Ambassador. Although, authentication and authorization is not their main focus and it's not readily available in their open source version.

Additional context

In our use case, we manage multiple clusters with identical architecture. We love the portability provided by kubernetes, and we value flexible solutions with completely automated deployment. We also prefer kubernetes-native solutions that use existing resources (Ingress) and annotations, or ones that create their own CRD with a well-defined API, over special snowflakes that use their own domain language or yaml in text fields.

[desimone]

Thank you for writing up this issue @fiahil ; Will investigate and hopefully have some updated thoughts on this soon.

\cc @travisgroth

[travisgroth]

Hi @fiahil, thanks for opening this.

This has been something on our radar but not fully fleshed out yet. Generally agree on making Pomerium as well integrated with Kubernetes as possible when running on the platform. To touch on the two choices you identified:

Use Pomerium as external authentication/authorization service for an existing ingress-controller (ingress-nginx, traefik, ambassador)

I don't think they have a standard auth check mechanism, so this could be problematic to support. I think the best approach is to have a Pomerium service that is responsible for proxying and doing authn/z. If there is a standard emerging, though, I think this is a conversation.

Use Pomerium as ingress controller by leveraging existing proxies or load balancers like Envoy

Currently Pomerium ships its own proxy. There's been some early discussion of whether that continues to be the case but for now it can/does do http itself. I'd like to set aside what proxy implementation is being used, since I don't think that is too important to achieve the desired outcome. Whatever the proxy implementation, we need something to observe ingress resources of a certain class, translate that into pomerium policy, and update Ingress resources as appropriate.

My inclination is to move this to a second process rather than try to bake it into Pomerium directly (so, the nginx approach rather than the Traefik approach). It keeps the critical path code simpler and doesn't conflate implementation of the various services with how they are configured. The responsibility for the configuration is handled at a higher level, and the proxy/authz/authn service only need to be aware of the config being updated.

So - what we need is a Pomerium ingress controller implementation. At face value, this isn't too hard. Pomerium's proxy layer, however, isn't as sophisticated as nginx, envoy, etc. We don't support things like dynamic SSL certs or the various configuration nobs and annotations that you might want on a full fledged ingress proxy...but I don't think that blocks a nascent ingress controller with some caveats around how it functions. We specifically support dynamic changes to policy configuration, which should be the core of changes we'd want to take in as an ingress controller.

What I see as next steps:

• Definitive call on in-process vs operator type implementation
• Spec of what can/will be supported. This is probably pretty brief, but it's nice to be explicit and set expectations
• Spec on how an ingress object signals policy controls

From there, work could zero in on implementation. The nginx ingress controller has some good info on what it is doing and obviously has source available.

[travisgroth]

@fiahil FYI - https://github.com/pomerium/pomerium/issues/425#issuecomment-567039741

[travisgroth]

@fiahil first RC of the pomerium operator has been integrated into our helm chart at https://github.com/pomerium/pomerium-helm/blob/master/charts/pomerium/README.md#pomerium-operator-experimental

https://github.com/pomerium/pomerium-operator#using
https://github.com/pomerium/pomerium-operator/releases

Not a full ingress controller at this time, as it leverages forward auth. The plan is to move towards a full ingress controller, but that depends on a few factors. Please report any issues or suggestions!

If you don't mind, we'll close this out and can shift further feature/improvement discussions into the operator repo.