Contour: Provide support for integrating with an external auth service.

[larkinkevin]

• Ambassador supports authentication via the "AuthService" manifest (and a user-provided external auth service that satisfies a very simple API).
• It appears that Envoy likes this approach, because they're planning to incorporate it: https://github.com/envoyproxy/envoy/issues/2828.
• If/when Envoy provides this functionality, it would be great if Contour could expose it to users.

[ihrwein]

I wanted to raise a similar issue then managed to convince myself that although this feature would be useful it's not aligned with this project's goals (note that I'm just somebody from the Internet who found this project some days ago.).

My reasoning is that different teams can have different authentication requirements. Contour is to have one Ingress that delegates the traffic to team-managed services. Those services can be simple services or API gateways that hides other services and those API gateways would be a better place for authentication. Just my two cents.

[larkinkevin]

I can certainly see where you're coming from @ihrwein. I just opened this issue to track the feature request (I figured I should at least make the ask).

Will totally understand if the Contour folks decide this is not a feature they want to support (because it's not handled by the Ingress spec).

[rosskukulinski]

Hi @ihrwein and @larkinkevin - thanks for opening this issue. I think this is an interesting feature request and something that could be a useful addition to the new IngressRoute CRD we're heads down on at the moment.

We haven't taken a close look at the APIs exposed via https://github.com/envoyproxy/envoy/issues/2828, but once it lands in an Envoy release, we'll take another peak.

[mattmoyer]

+1 for this feature. I'd love to have some Contour backends protected by the ext_authz filter in the style of an API gateway.

Envoy docs for the feature, since it doesn't look like those have been linked here yet:
https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/ext_authz_filter.html

[pims]

I’d be happy to contribute a PoC for ext_authz filter configuration if the contour team still believes it's a valuable feature to add.

Are you dead set on having this only available through the new IngressRoute custom CRD or should I also consider how to integrate it with the native ingress definition?

[rosskukulinski]

I think this may also be related to adding similar functionality as to https://github.com/heptio/contour/issues/68

[davecheney]

Removing the 0.11 milestone as external auth support is not available in a shipping envoy 1.9

[pims]

@davecheney Envoy 1.9.0 released on Dec 20, 2018 has support for ext_authz.

https://github.com/envoyproxy/envoy/blob/v1.9.0/docs/root/configuration/http_filters/ext_authz_filter.rst

Am I missing something?

[davecheney]

I’m sorry, that was my mistake. I confused this with another feature.

On 12 Mar 2019, at 10:18, Tim Bart [e-mail] wrote:

@davecheney Envoy 1.9.0 released on Dec 20, 2018 has support for ext_authz.

https://github.com/envoyproxy/envoy/blob/v1.9.0/docs/root/configuration/http_filters/ext_authz_filter.rst

Am I missing something?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[aslafy-z]

Any chance this goes back to 0.11.0 milestone @davecheney?

[davecheney]

I've readded the 0.11 milestone, but no promises, as usual, we're resource constrained and once we decide on a release date for 0.11, there may not be time to implement it.

[shabx]

Adding what customer shared in the ticket they have open for this issue.
They currently have teams using the NGINX External Authentication annotation that allows authentication against a pod running in GKP.
The documentation is at
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#external-authentication

[davecheney]

Due to the need to ship Contour 0.11 to address a security issue in Envoy 1.9.0 this issus has been bumped to 0.12.

@shabx this issue has been reassigned to 0.12, however I feel that its likely that when we rationalise the backlog for 0.12 (there is far too much in there for our team) this feature will be removed from the milestone until there is a design document for it.

[alexbrand]

Took a cursory look into what it would take to implement this. It seems like the external authorization filter can only be configured at the Listener level.

Given that we only have two Listeners (ingress_http and ingress_https) for all Ingress/IngressRoutes I _think_ this means that all Ingress/IngressRoutes would have authorization enabled? Furthermore, it seems like all authz decisions would have to be performed by the same external authz server?

[pims]

It can be enabled/disabled on a per route basis:
https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/ext_authz_filter

So a default could be set and overridden for each ingress.

[simonswine]

I've thought about that recently. I have a design PR in the works. Hopefully can share it later today.

Will update you here.

It uses a single auth service with context provided or disabled using the per_filter statement.

[alexbrand]

Yeah that was my fear. Doesn’t seem great to have a bunch of disables for all routes that are not auth. But maybe that’s the way to go.

The bigger concern in my mind is that all auth decisions have to go to the same server.

[simonswine]

In my design PR #1014 Contour would act as a single auth server and forward traffic to external services. Have a read of the draft and let me know what you think...

[travisghansen]

@simonswine I just read through the doc and think the external auth feature differs a bit from traefik and nginx etc. Instead of a signURL configuration option the proxy generally just returns the response from the auth service if it's not 2xx. This typically results in either a 302 with redirect or a 401 or some 50x. This design gives the auth service complete control over where to send someone and build powerful flows.

I'm currently working on a new project/service to do just that with oidc providers and have it working with both traefik and nginx. I'd love to get it working/test it with contour as well.

[travisghansen]

I lied, nginx is different (in a bad way). nginx is not directly passing through non 200 responses which is unfortunate. Both traefik and ambassador do this as described above:

• https://www.getambassador.io/reference/services/auth-service/#the-external-authentication-service
• https://docs.traefik.io/configuration/entrypoints/#forward-authentication

I'm looking into solutions that will work with kong and the ingress-nginx as my focus is primarily kubernetes ingress controllers (I use the term broadly here).

[davecheney]

I’m tentatively assigning this to 0.15 to take a crack at for Contour 1.0. At the moment we don’t have any resources available to work on this for 0.15 so it may get dropped.

[travisghansen]

As an update to my earlier comment my auth server is reaching a relative state of maturity at this point. I ended up broadening the scope to be a generic auth server so it has support for many plugins at this point (oidc/oauth2, jwt, ldap, htpasswd, firebase jwt, etc).

In addition to that a single configuration is setup as an auth pipeline so that the same service can be serviced by several mechanisms (ie: ldap and oidc). You can even target specific plugins in the pipeline based on the nature of the request...for example if the request has an authorization header don't continue on to the oidc process..things like that. For my personal setup I'm doing just that with ldap basic auth and oidc backed by the same ldap store (keycloak). This way end-user usage is SSO and yet I can still do machine to machine stuff via basic auth.

A single deployment can handle any number of services/configurations given the unique approach to configuration. Meaning, you don't have redeploy the app over and over to cover as many services as you want.

A powerful assertion engine along with configurable header injection for consumption by backing services is also in place (ie: auth with oidc and pass the email down as a specific header value to be used by grafana's external auth setup). Or authenticate against GitHub but only allow access if the user is part of a given organization..things like that.

A helm chart is available in the repo for easy installation.

All of this of course depends on a reverse proxy that supports the general pattern of external/forward auth.

https://github.com/travisghansen/external-auth-server

[davecheney]

Hello,

I want to give an update from this issue. As mentioned in https://github.com/heptio/contour/issues/432#issuecomment-503014473 the assignment to 0.15 was tentative and during the cycle noone was available to work on a design for this. As 0.15 is our last development cycle before the 1.0 freeze I am sorry but this feature will not make the cut for Contour 1.0.

I am moving this to the unplanned milestone to indicate we will revisit it after Contour 1.0 ships in November.

[victornoel]

Any news on this? For the record pomerium is seriously considering supporting forward auth (see https://github.com/pomerium/pomerium/issues/94#issuecomment-535335159), this would be a great combination with contour :)

[davecheney]

This issue is currently on the backlog. We’ll be doing a pass over the backlog once contour 1.0 is released.

[imranismail]

Any update on this, I'd be happy to contribute a POC

[clmssz]

Hi folks, any news on this ?

[malyabee]

any update on this?

[youngnick]

Hi everyone, thanks for all your pings. This issue is still on the backlog at the moment, as it needs some design work before we can implement it in the way we want to.

If you are interested in this issue, the thing that we will really need to be able to do a good design is to get an understanding of what you are looking for with external auth. I'm thinking some things like:

• What sort of auth do you need? Just protecting something with Basic auth? OpenID? Something else?
• What do you want to connect to? Where do you want the auth to come from?
• Anything more general you can tell us about your use case is helpful.

We can definitely build something without this information, but your help can make it much easier.

[derekperkins]

Personally, I'm just looking for Envoy external auth to be exposed without any extra frills. On inbound requests, check against an http/grpc service for pass/fail auth.
https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/auth/v3/external_auth.proto

I think that separate from this issue, there are specific auth implementations like basic, OpenID, jwt authentication, etc. that would be nice to have available by default, but those can all also be implemented with external auth in the meantime.

[travisghansen]

Yes exactly. This implementation makes those questions irrelevant (from the context of contour).

[youngnick]

In general, we have a couple of rules when Contour is exposing Envoy settings:

• We try to ensure that there aren't any ways to shoot yourself in the foot with the settings if we can
• For values where there are reasonable defaults, we just set the defaults, and see if there is anyone who needs them changed.
  (I'm working on writing this statement up in a general way under #2224 )

In this case, although the code to expose the feature seems like it should be reasonably straightforward, we need to ensure that the way we implement it fits with those goals, which means spending some time reading and understanding the Envoy docs, and what the feature is trying to achieve. I can understand how this can be frustrating if you've already done that and just want to twiddle the knobs, but we have to look after everyone else who is using this project too.

The things that I want to understand before we look at landing this feature are:

• What's the support contract? What happens when the gRPC auth service is unavailable? Should Contour be checking the status and reporting it? This is particularly important given Contour's shared-ownership model, where the team that is implementing the auth may not be the team that is actually serving things.
• In terms of settings, when we specify the config for the service, what are the settings that need to be passed and what should the defaults be?

The gathering of use-cases is just to ensure that we're hitting everyone's use case as part of this process.

[stevesloka]

I think we could follow much of the design for rate-limiting which I never finished the final PR for, but it followed what @derekperkins mentioned, just exposing the gRPC endpoint.

I think we just need a new design PR for this service to answer the questions @youngnick pointed out.

[jpeach]

I haven't investigated this issue, but a quick extemporaneous comment. There must be existing external authentications for envoy and for other ingress controllers that expose envoy external auth in some fashion. So when we think about Contour support for this feature, we should keep in mind how these existing authentications services could be integrated.

[travisghansen]

I have sample configs here for ambassador and istio if it's helpful to have an idea how it would look..

https://github.com/travisghansen/external-auth-server/tree/master/examples

[youngnick]

Thanks @travisghansen, that's quite helpful.

[travisgroth]

Here is the Pomerium documentation on external/forward auth configuration: https://www.pomerium.io/configuration/#shared-settings

It has allowances for both the nginx and traefik/ambassador implementations.

What's the support contract? What happens when the gRPC auth service is unavailable?

In this case, the proxy should deny requests (fail closed) to the protected URLs. It should not impact unprotected routes.

In terms of settings, when we specify the config for the service, what are the settings that need to be passed and what should the defaults be?

Some considerations that come to mind:

• Be able to specify the original request headers that are passed back to the auth service. Defaults should include setting sufficient headers to inform the auth service of the original request provenance (including cookies). I believe Traefik uses X-Forwarded-Uri and nginx uses X-Original-Url.
• Pass cookies set by the auth service back to the client to set session tokens. This should default to on, I think.
• Option to follow auth server redirects to a sign-in page. Nginx requires this as a secondary configuration and it carries some operational overhead that is better avoided. Again, default to on. Without it, there will need to be another way to configure contour for sign-in pages.

Hopefully that helps. Happy to answer any questions.

[youngnick]

I really appreciate all the feedback on this issue, and I understand that everyone is keen to see it. I'll update again when we've been able to prioritise this work.

The next steps from here are:

• A design document addressing all of the info here
• Once that is merged, we can start thinking about implementation.

[ryanelian]

Need basic auth in Contour to protect services without authentication WITHOUT using NGINX between the Contour and the actual app...

[vukg]

@ryanelian +1

[jpeach]

Please have a read of https://github.com/projectcontour/contour/issues/2459 and leave any comments you have there. I'd like to get consensus around the direction and scope before proposing a concrete design.

Thanks :)

[ryanelian]

Sure. I just posted my take on Basic authentication in HTTPProxy CRD in that thread. I hope that proposal will be incorporated into the final product.

[tomqwpl]

In terms of JWT authentication, Envoy directly supports verifying JWT tokens, without going to an external service. This is nice for performance, avoiding the overhead, on every HTTP request, of making another HTTP request to verify the auth. The nginx plus ingress controller also supports this method (direct JWT verification). Personally I'd like to see that as an option here, and not have to rely on a generic external auth endpoint through which all requests are routed.

[bitva77]

No pressure or anything but I just want to say that this feature lacking will keep us from using Contour. We currently use Nginx for external auth to a service via nginx.ingress.kubernetes.io/auth-url.

This looks to be the only major feature missing from a company like ours from adopting Contour as our ingress controller standard. Using projects with official CNCF love is very important to us (and I'd imagine other enterprises).

:)

[jpeach]

No pressure or anything but I just want to say that this feature lacking will keep us from using Contour. We currently use Nginx for external auth to a service via nginx.ingress.kubernetes.io/auth-url.

This is coming though it won't be a drop-in replacement for nginx auth; take a look at #2643.

[jpeach]

Blocked by:

• [x] Auth: Add ExtensionService CRD #2710
• [x] Auth: Enable ExtensionService CRD #2711
• [x] Auth: Add an Informer to watch ExtensionServices #2712
• [x] Auth: Create Envoy clusters from ExtensionServices #2713
• [ ] Auth: Watch ExtensionServices availability and update Status conditions #2714
• [x] Auth: Add HTTPProxy authorization fields #2715
• [x] Auth: Add external authorization documentation #2716
• [ ] Auth: Add external auth integration tests #2717
• [x] Auth: Set ExtensionService status Conditions #2874
• [ ] Auth: Support external services in ExtensionService #2875
• [ ] Auth: Add metrics for ExtensionServices #2889

[manast]

In terms of JWT authentication, Envoy directly supports verifying JWT tokens, without going to an external service. This is nice for performance, avoiding the overhead, on every HTTP request, of making another HTTP request to verify the auth. The nginx plus ingress controller also supports this method (direct JWT verification). Personally I'd like to see that as an option here, and not have to rely on a generic external auth endpoint through which all requests are routed.

That seems like a different feature, or if you like a refinement that can be implemented later. As for now, the external authorization even if not optimal in all cases would help in solving many use cases that are kind of impossible to achieve today.

[manast]

@jpeach Should I open a separate issue for implementing external_auth on TCP or would this one include it already? And close this one in such case https://github.com/projectcontour/contour/issues/2855

https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/ext_authz_filter#config-network-filters-ext-authz

[jpeach]

@jpeach Should I open a separate issue for implementing external_auth on TCP or would this one include it already?

@manast Yeh, could you please file a separate issue? This is scoped only to HTTP.

[manast]

@jpeach ok, done: https://github.com/projectcontour/contour/issues/2888

[sunjayBhatia]

Closing this issue as we have linked more granular issues open