Opa: OPA - MongoDB query translator

[ashutosh-narkar]

MongoDB is a general-purpose, document-based, distributed database with a rich query language. OPA features a high-level declarative language called _Rego_ purpose-built for expressing policies over complex hierarchical data structures. OPA is often used to enforce policies over incoming API requests, but using OPA's "partial evaluation" feature it is also possible to enforce policies when data is accessed inside of document-oriented databases like MongoDB. In this model, callers query OPA to obtain a set of conditions to apply to the database query and then rewrite the database query accordingly. There are existing projects that translate "partial evaluation" results to SQL and Elasticsearch. This project would involve designing and implementing a Rego to MongoDB query translator that supports basic relational operations like ==, !=, >, <, etc.

[kushthedude]

Hello @ashutosh-narkar, I am aiming for GSoC 2020 with OPA Project and I would like to get started with the project and I have already set up a local go development environment. Any help regarding how can I make my first contribution to the OPA would be really great.
PS: I am new to Go but I have quite an experience with Python.

[ashutosh-narkar]

Hello @kushthedude ! Thanks for your interest. Refer the OPA docs to get familiar with the project. I would recommend looking at the low-hanging issues to contribute to OPA.

[kushthedude]

Hello @kushthedude ! Thanks for your interest. Refer the OPA docs to get familiar with the project. I would recommend looking at the low-hanging issues to contribute to OPA.

@ashutosh-narkar Thanks for the info, I have setted up the development environment on my local and will soon start working on some issues .

[sresthas]

Hey, If nobody has been working on this, can this be proposed for CommunityBridge?
I would be happy to work on this

[ashutosh-narkar]

Thanks @twisted-sres for your interest ! We've proposed this project for CommunityBridge. PR here.

[VineethReddy02]

@ashutosh-narkar I have a couple of questions & thoughts here

1. So why are we specifically targeting MongoDB?
2. Do we have any advantages of using MongoDB over the traditional approach?

Why shouldn't we consider a gRPC plugin-based architecture were OPA works as gRPC client and sends out the rego queries to a gRPC server here the logic of query translation resides. By this model, we are building an extensible pipeline for OPA to integrate with the desired database by implementing the gRPC server logic out of the OPA tree for the desired DB. In the future, this helps us to extend the support for any database out there and maintenance of DB specific query logic also resides out of OPA.

I would appreciate your thoughts & suggestions here.

[ashutosh-narkar]

Thanks for your interest @VineethReddy02 ! MongoDB is a popular database and we already have ES and SQL integrations, so we proposed MongoDB. I haven't explored the gRPC approach and it would be good to have more details on that. OTOH, having the library OPA integration has its own advantages like low latency etc and since MongoDB has a Go driver we can leverage that as well.

[Syn3rman]

Hey @ashutosh-narkar,

I'd like to work on this idea under community bridge, so could you tell me whether opa has a slot for q2 2020?

I've gone through the documentation and the blog posts, and I'll start working on some issues tomorrow to get familiar with the codebase.

The implementation would be similar to the rego to sql converter, right?

[sresthas]

@Syn3rman Yes, OPA has a slot for q2 2020. I am also planning to apply for the same.

[ashutosh-narkar]

Hello ! We are accepting proposals for this project. You can submit a proposal here. Thanks !

[Syn3rman]

Hey @ashutosh-narkar,

I am planning to start working on my proposal for this idea, so is there any template that we have to follow?
Also, is this a good place for bouncing off ideas related to it or should I use slack channels/dm instead?

Thanks!

[ashutosh-narkar]

Thanks for your interest @Syn3rman ! There isn't a template as such. At a high-level a design proposal covering as much detail as relevant from the perspective of the project would be a good start.

Any mode of communication works for me.

[aribalam]

Hi @ashutosh-narkar !
I am a third-year student from IIT Kharagpur, willing to contribute to this project under the Linux Community Bridge program. Could you point me in the right direction on how to get started?

Also, is there a way to communicate with you personally (Like slack, mail etc.)?

[ashutosh-narkar]

Hello @aribalam ! Thanks for your interest ! You can submit your design proposal here. You can post your questions here or reach out on OPA slack(@ash).

Check out the OPA docs to get familiar with the project.

[ashutosh-narkar]

Hello all,

If you would like to submit a proposal, please see these guidelines for the application process.

Thanks
Ash Narkar

[ashutosh-narkar]

Hello all,

A gentle reminder that May 18th 2020 is the last day to submit proposals for this project. See important dates here.

Guidelines for the application process can be seen here.

Thanks
Ash Narkar

[agarwalrounak]

Hi @VineethReddy02!
I had prepared a proposal for this project: https://docs.google.com/document/d/1Pq1eYOXrGeoJj5M_8Kup_WJAp_0EkhSipGU_QvMn660/edit#
I hope it will be useful to you. Please let me know if I can be of any help.

[VineethReddy02]

Hi @agarwalrounak
Good to see your proposal and our proposals share common grounds in detailing the context & requirements. Here is my proposal
I appreciate you for sharing the proposal. :)

[rfielding]

Thanks for your interest @VineethReddy02 ! MongoDB is a popular database and we already have ES and SQL integrations, so we proposed MongoDB. I haven't explored the gRPC approach and it would be good to have more details on that. OTOH, having the library OPA integration has its own advantages like low latency etc and since MongoDB has a Go driver we can leverage that as well.

I have my own LISP language that is doing something similar to OPA, I am also looking at just moving the evaluation into MongoDB itself. Doing this as a remote-service is not going to work for the most important use cases that I have. Say that the whole point of what you are doing is to CALCULATE permissions and security labels on object; not simply to boolean filter them in/out of a query. A user comes in, and has JWT claims that sign off on the user's email address, the groups he is in, etc. These claims were digitally signed, so that we can believe that they are actually true.

• Take JWT claims (a json value) as an input to the query, along with normal query parameters.
• Every potentially returned object has a function on it... OPA code
• Of the tens of thousands of potentially returned results, you want a few hundred matching results returned. This motivates you to do this calculation in the database itself. You may not even trust the application to evaluate rows to be filtered out.
• The opaEval(jwtClaims) literally calculates permissions from the claims, where the evaluation function is per-record. For example, imagine that it calculates access attributes, and labels that are associated with the return value: {read:true,update:true,label:"SECRET//Aliens",labelBackground:"red",labelForeground:"white"}
• Without read access for this record, it should be filtered out of the query result entirely. But if it comes back, it's plausible that a listing of what should be ALLOWED, and any handling requirements for the user can be returned. Opa lets you take in a JWT claim as input, and calculate an arbitrary json field as output.
• The excellent thing about using a language function attached to each record (ie: like OPA) is that the user only needs to make sure that policies match JWTs.

Doing this remotely to a GRPC call is ok for some use cases, but for filtering queries; it's exactly what you never want to do. This evaluation needs to be done in the app before it returns results, or ideally in MongoDB itself.