Opa: Feature request: improved support for external data source incorporation

[anadon]

I need to create a security project to work with actively in development code over several interdependent projects running in kubernetes mainly using python 3. What each of these are guaranteed to get is a JWT. One of them is responsible for JWT creation which gets passed around. There is PostgreSQL DB and a CockroachDB which store our business and customer data.

What I'm designing has as its goal is to be conceptually concise, elegant, fool-proof, etc API to add to these projects. Other people will be using this, and the more it gives a consistent and simple API interface, the better. What I would like to do from the user side is just make a REST call with the JWT to describe the particular situation that is going on in the code.

Getting more real, we need to make trivial wrappers for each of our calls. If on a query we get a 404, we are going to reload the projects rules to the instance of OPA running in a service via OPA's REST API. This is to correctly handle updates which can happen and not worry about restarting the service for unloading and reloading the rule. It helps with horizontal scaling. I would love to keep the wrapper this simple so I can just use a decorator around a 1 line function. If I have to add needed context to provide to OPA, the logic complexity dramatically increases with no increase in functionality.

In order to do this, all of the security policy logic must be self contained OPA side. This means security logic and lookups for that security logic would be in OPA and data in a set of DB's rather than security logic in OPA, data lookups for OPA in each individual project, and pulling those data sources from various DB's and run state from projects to feed into OPA with an inconsistent argument list. Splitting data to feed into OPA means that changes to OPA policy configuration must be synchronous in the calling project's logic and OPA's policy configuration.

All this establishes what callers would like as a call interface and why, but not so much what OPA's internal configuration _stuff_ would look like. Calls to different DB's have interfaces that are consistently almost compatible. I think there ought to be two approaches to this. First, the overly functional and discouraged system shell result from a sprintf like call. This allows not the worst handling of sensitive data and a flexible interface to get results in tight scenarios, but is typically a terrible idea to use. The next would be a variety of suitable calls into various DB implementations. Our highest priority DB interface is cockroachDB. I'm not sure how a persistent connection would work with OPA, but self contained connection and query would be either a file used to connect or username password pair, location of the DB as a URL, and then the query text. Of course this will vary between DB implementations, since sqlite would be a file path and a query. Sqlite3 might be a good first target for inclusion over more full-featured offerings. This may also be the opportunity to figure out how to make a general external data source plugin interface, so other data sources like ldap could get their own query support, or if someone needs to make something crazy and off the wall like a homebrew RFID door unlocking module, they could.

Let me know what other input might be helpful.

[anadon]

As an aside, I think there may be something to the local context of the project and the global context in the DB. Getting data from calling code or a project is local and generally more fragile. DB tables are fairly steady and reliable and a better idea to rely on. Mixing the two with local context put into a global context can be fragile, and putting global context into local context comes with extra difficulty.

[srenatus]

I had brought up a similar thing some time ago, here in slack:

I could see some use in having a plugin that would query postgres with a set of configured queries that would return some JSON like

path          | json
--------------|------------------------------------------------
data.policies | {"id1": { "role": "admin", ... }, "id2": ... }
data.roles    | {"admin": ["alice", "bob"], "ops": .... }

these would then replace the store’s content.

adding a notification channel would allow signalling to the OPA plugin that there’s been an update to the data it cares about, causing a refresh. (as an alternative to updating periodically)

However, I notice that this would be quite different from the _expose querying data sources from policy rules_ approach… is it a viable alternative, or an instance of an approach you would advise against?

The trade-off highlighted in the thread was that typically, you'd differentiate _querying the data source_ vs _pulling in data from it_ by the amount of data: It might not all fit into memory, so you might have to query.

I thought this might have some relation to what you've outlined (if I've understood that correctly). 😃

[anadon]

I'd consider each access of a variable or external retrieval a 'query' or 'access' if we want to go with a mathematical usage. I don't see a significant difference between the two. But I know I need external data sources.

I'd like to think any action has the following: Actor, Action, Object, Actor Local Context, Object Local Context, and Global Context. Any full featured security system will work with these. Right now, OPA needs work on working with Global Contexts -- things globally accessible to all parts of a system or program, like a database.

[anadon]

If you're looking for testability, I would suggest a coding best practice to have an input stub, which gathers all of the information, pass it to a separate rule or function, then have the logic contained in that back end rule or function. Maybe in functions, have default values which are query lookups.

[tsandall]

@anadon thanks for filing this. OPA currently has very primitive support for accessing external context via http.send but beyond that, there is not much. You could implement your own built-in functions that fetch data from external sources or provide your own implementation of the OPA storage layer, but that's a decent amount of work.

One thing I've been wondering about lately is whether we should define a standard interface for fetching external/remote context...then you could configure OPA with your context endpoint, and references to that context would trigger RPCs. Something like...

OPA configuration:

services:
  controller:
    url: https://example.com/opa-control-plane/v1
bundle:
  name: app_policy # download app policy bundle
remotes:
  catstore:
    url: https://example.com/catstore/v1

Then you could refer to remotes inside your policy:

package example

allow {
   input.method = "GET"
   input.path = ["cats", id]
   cat_context := remote.catstore[id]  # refer to remote "catstore" datasource outside of input/data
   input.subject = cat_context.permitted_users[_]
}

The remote would be responsible for implementing an interface that allows callers to access hierarchically structured data:

// Returns a value at path.
Get(path string) (interface{}, error)

// Returns list of keys under value at path.
Keys(path string) ([]interface{}, error)

[anadon]

Possibly. I'm not sure which approach would end up being most efficient. I could get either to work.

[tutman96]

What I am currently working on right now is using graphql to query my external sources of data. I am writing a small service that sits next to OPA that makes these queries and uses graphql subscriptions to get notifications when the data changes. GraphQL works really good for our use case because it allows us to only populate the data we care about in the policies. It would be awesome if the graphql community would finish their standard on live queries (queries that update themselves “magically” as this would remove a lot of code from our service).

The service is being designed to support multiple sources of data (or bundles for that matter) and will keep OPA up to date as real time as possible. This service will later be rewritten into an OPA plugin once I get all the small details worked out outside of it.

[prahaladdarkin]

@tsandall @anadon Was this issue resolved through a functionality addition in OPA? I have a similar use case where-in I would want to read data to be evaluated from SQL rather than store them up in memory.
@tsandall I also agree with the fact that there should be a unified interface driven mechanism that allows clients to specify the store either through environment variables or request parameters

[tsandall]

@prahaladdarkin this issue has not been resolved. Over the last year the http.send function has matured however there are still improvements that could be made. We haven't done additional work in this area aside from showing how users can _extend_ OPA. I have heard of some users customizing OPA with logic that fetches data asynchronously and caches inside OPA's in-memory store. I've also heard from some users that have extended OPA with custom built-in functions to fetch data from non-HTTP based stores.