Hydra: Kubernetes Mode

[kminehart]

Is your feature request related to a problem? Please describe.

I've been considering something like this for a while. It hopefully solves a couple of issues in the Kubernetes and "cloud native" space.

Right now, when using Hydra in a Kubernetes environment, the process of deploying Hydra to Kubernetes has a few issues (like any app with state):

1. It's a bit of a pain to create the Hydra pods, and then use kubectl exec or curl to create / authorize clients.
2. A dependency on a RDBMS like Postgres or MySQL makes Hydra less suitable for a dynamic cluster environment, and makes automatiic horizontal scaling hard to achieve appropriately.

Hydra naturally introduces some level of state when being deployed, which is often not suitable for a Kubernetes environment.

Describe the solution you'd like

• Add a new flag for Hydra: -kubernetes (or set DATABASE_URL=kubernetes)

When -kubernetes is enabled, Hydra would essentially become a Kubernetes controller:

• Hydra will synchronize CustomResourceDefinitions on the runninng cluster for HydraClients, or other declarative API objects.
  
  
  
  • This would allow creating a new Client via kubectl create ..., which would persist restarts of Hydra.
  
  
  • Another OIDC / Oauth2 Provider called "Dex" uses a similar system for storing state, which you can read about here: https://github.com/dexidp/dex/blob/master/Documentation/storage.md
  
  
• Hydra will subscribe to Kubernetes API events, and update its storage to reflect client updates done via kubectl

Additional context

There would need to be more discussion on things like:

• Handling state that is not clients, like the tables in consent/migrations.
  
  
  
  • How would having potentially multiple "replicas" of Hydra work without central storage like postgres? (Maybe something like an embedded gnatsd server, or https://github.com/hashicorp/memberlist
  
  

[aeneasr]

I generally like the idea. One centerpiece will be to figure out how we can store access/refresh tokens in a scalable way - I'm not sure CRDs are the right place for that. Another idea I had was to import specific data on boot from disk. I'm not sure if that's possible, but it would allow to mount CRD/whatever on disk and hydra would load that on boot. This would not work with watching though. The problem with hydra really is that clients are usually created from 3rd parties without access to the k8s cluster. I'm not sure how much optimizing we should do towards k8s in that regard. The same goes for consent/login flows which are always browser-initiated. The only thing that's really super static is usually the JWKs.

If we're talking Oathkeeper or Keto, there it really makes sense because rules etc. are rather static and there a watcher for CRDs is really smart imo.

[kminehart]

One centerpiece will be to figure out how we can store access/refresh tokens in a scalable way - I'm not sure CRDs are the right place for that

I agree with that.

CRDs are particularly good for objects / resources which humans have to modify / create manually.

There's a pretty good section here about that: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#should-i-add-a-custom-resource-to-my-kubernetes-cluster

This would not work with watching though. The problem with hydra really is that clients are usually created from 3rd parties without access to the k8s cluster

I actually didn't think about that use-case, but it definitely makes sense, and I imagine that would be more common than the one I'm about to discuss.

In a previous scenario I've run into, we had a fairly large list of authorized clients that were all internally maintained, but changed infrequently. CRDs would have been ideal for us then.

It definitely makes a lot of sense for Oathkeeper and Keto.

I built something similar for Ladon a while ago which would watch for LadonPolicies and update Redis to reflect the LadonPolicies created in k8s. This was really helpful when we were defining accounts "default" users for each service as it was just some extra yaml that went along with the Deployment.

[aeneasr]

Yeah that makes sense, Hydra is definitely being used in places where only 1st party clients are being deployed. Maybe we could watch CRDs (could this be a small library?) and simply upsert the existing datastore on changes/inserts?

[kminehart]

Makes sense to me. We'd also have to remember to watch for deleted resources as well.

It could be a small(ish) library, but will have a big dependency on https://github.com/kubernetes/client-go.

[aeneasr]

Ah yeah, that definitely looks like a complex dependency, maybe there is a lightweight version of that for CRDs?

[kminehart]

My gut says no, probably not. If you have time you might want to take a look at the sample controller:

https://github.com/kubernetes/sample-controller

A common pattern I've frequently seen is a separate controller server. A hydra-controller or ory-controller in this case might watch the Kubernetes API, keep the CRD definitions synchronized, and send API requests to Hydra as the CRDs are updated / created / deleted.

[aeneasr]

Oh yeah, that makes much more sense!

[MOZGIII]

Is there any progress on this one?

[aeneasr]

Watch out for:

• https://github.com/ory/oathkeeper-k8s-controller
• https://github.com/ory/hydra-k8s-controller

The awesome guys from Kyma / SAP are taking over the implementation of that.

/cc @piotrmsc

[aeneasr]

Closing, see maester projects!