Hydra: RFC: k8s and microservice mesh sidecar pattern support

[ermik]

Discussion should cover the details of running Hydra & its peers in a microservice mesh environment as can be seen in the example of istio on Kubernetes.

Abstract

In a microservice environment, to decouple the implementation of individual services and offload the security handling to a third-party actor a concept of mesh is used. Mesh governs the communication between microservices and secures it with TLS, provides functionality for canary deployments, circuit breaking, service discovery, tracing, and more. It operates on L7-L4, with strong bias towards L7.

Such environment can be achieved in one case by use of a proxy _sidecar container_ that forces all inbound and outbound traffic through itself. This means the service being ran does not ever talk to any other service, let alone public Internet, directly. The service talks to localhost over HTTP. When exposed to the public web, the TLS is terminated at the cluster ingress resource (NB: does not have to be of k8s type Ingress), which also serves the required TLS certificates (which at times are minted automatically).

Opinion / Learnings

Personally I see much benefit in using a mesh, including in particular the case of running ory/Hydra behind a strong set of policy directives and security features. This includes a case of ambassador proxy being used to authenticate the traffic before it gets to the mesh, using ory/Oathkeeper service inside the mesh, with Oathkeeper interacting with Ory ecosystem solely within the mesh using internal Kubernetes DNS resource notation e.g. hydra.auth.svc.cluster.local. Additionally in our case there is a service that enacts the OAuth2 flow on behalf on the mobile client without leaving the mesh which raises the need for running serve web configured to look for login & consent endpoints, callback endpoint and "self" endpoint using HTTP and aforementioned internal resource names.

Known issues:

• HTTPS_ALLOW_TERMINATION_FROM environment variable set to 127.0.0.1/32 should have been enough to allow sidecar interaction, but in case of Envoy proxy (which is what istio sidecar is) we've seen malformed X-Forwarded-Proto header (set to http regardless of downstream rules) which forces us to use --dangerous-force-http
• Hydra does not allow non-localhost redirects to have http:// scheme (fosite): https://github.com/ory/fosite/blob/fa654089e09900d842117827ec2f6258323ec436/handler/oauth2/flow_authorize_code_auth.go#L74

Points of discussion

• Sidecar pattern produces requests originating from localhost & its subnet — would it be safe to assume that localhost <-> localhost communications deep inside the cluster with an optional addition of mutual-TLS between cluster components are as secure as publicly available https?

[aeneasr]

Sidecar pattern produces requests originating from localhost & its subnet — would it be safe to assume that localhost <-> localhost communications deep inside the cluster with an optional addition of mutual-TLS between cluster components are as secure as publicly available https?

Yes, it's as secure. The question I have if this is an issue with istio or hydra.

[ermik]

It is both. I assume that HTTPS_ALLOW_TERMINATION_FROM would be enough to achieve a secure environment as long as the Envoy issue with XFP headers is fixed. But past that, there are questions about what affordances could be made when the internal environment assumes TLS termination (such as allowing http:// redirect to resources that are not called "localhost")

[aeneasr]

The point of being super strict about this is that HTTP is not a secure protocol. The security model OAuth2 is built upon is TLS, it does not have any other type of security mechanism - in contrast to OAuth (1.0) which used e.g. HMAC based message authentication. Giving people slack on this is an explicitly bad concept for this product. I know that TLS is perceived as being hard (it's not that difficult imo), but so is OAuth2, and systems design in general. I'm not sure how a compromise would/should look like here but I'm - as always - open for suggestions.

Regarding your question, of course is mTLS more secure than public TLS because both parties are authenticated. If mTLS is activated and used for HTTP communication, the proxy should set the right forwarded-proto, as do proxies such as ngnix, kong, aws api gateway, ...

[ermik]

I will try to get my thoughts together and stub an additional section to hydra Production use document in the docs that will cover this.

[mabushey]

Please do not pay any attention to the Istio docs that talk about mTLS. The first step to getting Istio to actually work (and it does work well) is to disable mTLS.

[ermik]

@mabushey I've never disabled mTLS and Istio always worked for me, since v0.6 or so. Let me know if I can help with anything Istio related. This issue, while certainly under the radar at this point, was mostly to bring attention to deployability of hydra in a k8s environment which carries some assumptions different from the public web regardless of usage of additional tools and services such as Istio or Linkerd.

[mabushey]

@ermik
I've been using istio since .8 and have never gotten mTLS to work until today on 1.1. I'm now using the banzaicloud istio-operator which is awesome and specifically disables mTLS by default. I need to get something working to provide JWTs for auth access for internal services (ie jaeger) in the cluster. The Ory tools look really good from the standpoint of being written in go and therefore having tiny container RAM requirements. Ory has a blog about using oathkeeper with Ambassador (another envoy proxy "system") but I'm having issues trying to adapt it to istio. I've gotten this far:

✗ curl https://ok.green.example.com/health/alive 
{"status":"ok"}%         

✗ oathkeeper rules --endpoint https://ok.green.example.com import access-rule-oathkeeper.json 
Importing rule oathkeeper-access-rule...
Successfully imported rule oathkeeper-access-rule...
Successfully imported all rules from access-rule-oathkeeper.json%

 ✗ oathkeeper rules list --endpoint https://ok.oro.cyndx.io                              
[
        {
                "authenticators": [
                        {
                                "config": null,
                                "handler": "anonymous"
                        }
                ],
                "authorizer": {
                        "config": null,
                        "handler": "allow"
                },
                "credentials_issuer": {
                        "config": null,
                        "handler": "noop"
                },
                "id": "oathkeeper-access-rule",
                "match": {
                        "methods": [
                                "GET",
                                "POST",
                                "PUT",
                                "DELETE",
                                "OPTIONS",
                                "HEAD"
                        ],
                        "url": "https://ok.oro.cyndx.io/\u003c.*\u003e"
                },
                "upstream": {}
        }
]

I cringe quoting Istio docs, but they show this as being how you use a JWT for auth:

origins:
- jwt:
    issuer: "https://accounts.google.com"
    jwksUri: "https://www.googleapis.com/oauth2/v3/certs"
    trigger_rules:
    - excluded_paths:
      - exact: /health

✗ curl https://ok.green.example.com/oauth2/v3/certs 
404 page not found

[ermik]

@mabushey if you are using a reverse-proxy, such as Ambassador, it will most likely terminate TLS. After this point it's a matter if using istio-specific Ingress resources to send the insecure internal traffic to. If you won't terminate TLS, your requests will need to be authenticated by the means of mounting istio-certs in ambassador and using them to sign the cluster-bound requests. Oathkeeper can tell istio to block certain requests from passing through Ambassador, but it has no effect on istio operations. Finally, if you wish istio mesh to check JWKs for you, rather than implementing those controls inside the individual services (your services always require authentication) you would enable token introspection on jwksUri accessible from within istio-system namespace and authenticate requests before they hit those endpoints. If they route through ambassador make sure headers are preserved so that istio can check the token validity.

[mabushey]

Oathkeeper can tell istio to block certain requests from passing through Ambassador

I'm not using Ambassador (1,764 Github stars), I'm using the much more popular Istio (16,475 Github stars) service mesh. They both use Envoy as the sidecar proxy. I fail to see how it would make any sense to use two routing frameworks that utilize the same proxy engine. I think I'd need a quantum computer. ;)

[aeneasr]

Ambassador is a proxy on top of Enovy...

[mabushey]

@aeneasr Exactly, Ambassador and Istio are both proxies/service meshes that use Lyft's Envoy.

[aeneasr]

A service mesh is a framework for running microservices. A proxy (or API Gateway) does routing and stuff that API Gateways do. That's a serious difference. By the way, Oathkeeper works with Envoy as well since they've added auth filters based on HTTP afaik.

[mabushey]

@aenesr OK, Istio is a service mesh framework that incorporates Envoy as a sidecar/proxy. Ambassador's features are a subset of Istio that include routing also using Envoy. The point is Istio does everything Ambassador does using the exact same proxy engine. In no way does it make any sense to use both together. Ambassador is like an Istio lite. Istio has 10 times the user base of Ambassador.

Glad to hear Oathkeeper works with Envoy. :)

[mabushey]

I did some Googling for the hell of it. Weird results. Sounds like the Ambassador people like the concept of using Istio in addition in Ambassador, while the Istio people think this is stupid. Makes sense as there's nothing that Ambassador can do better than Istio except be simpler and easier to learn.

[wmorgan]

Let me know if there's anything we can do to help from the @linkerd side! (cc @grampelberg)

[alQlagin]

any updates? 🤨

[aeneasr]

Regarding which of the mentioned issues?

[aeneasr]

Parts of this will be resolved as #1962

[ermik]

istio/istio#7964 also had some traction in recent months, which also helps for that particular mesh implementation (envoy); I have not had a chance to test yet thou