Opa: Add variable hierarchy for policy severity/enforcement

[phenixblue]

Expected Behavior

Add mechanism to allow for a variable approach to policy enforcement (from passive to fully enforced).

Something like a globally defined DENY Level (configurable as LOW, MED, HIGH) that sets a threshold that is inversely related to a "severity" definition for each individual policy.

| Deny Level | Severities Levels Blocked |
|--- |--- |
| OFF | None |
| LOW | HIGH |
| MED | HIGH, MED |
| HIGH | HIGH, MED, LOW |

This coupled with an alerting mechanism can allow for a lot of flexibility for users to ease from an inform stance to full blown enforcement, or the ability to have the same set of policies defined across many environments (Development, QA, Production, etc.), but have different levels of enforcement.

Actual Behavior

NA

Steps to Reproduce the Problem

NA

Additional Info

[tsandall]

@phenixblue thanks for filing this. If I understand correctly, this is about controlling how the policy decisions are handled at the enforcement point, e.g., does the resource get denied? does it just trigger an alert? does it do both?

Today most admission policies are written as "deny" rules in OPA. However, "deny" is not special in that it's just a name in the policy. We could change the name to "violation". As long as we update the policy (or the caller if they're explicitly querying "deny") the result would be the same.

violation["container missing requested memory"] {
  containers[c]
  not c.resources.requests.memory
}

Now the question is how to deny and/or alert when there's a violation.

To do this we could have rules like deny and alert.

deny[x] {
   violation[x]
}

alert[x] {
   violation[x]
}

The controller would query deny to decide whether to block the resource and it would query alert to decide whether to alert on the resource.

With this convention we could fine-tune deny-vs-alert based on environment.

# Deny resources with violations in the production namespace.
deny[x] {
  input.request.namespace = "production"
  violation[x]
}

# Alert on resources with violations in non-"sandbox" namespaces.
alert[x] {
  ns := input.request.namespace
  not data.kubernetes.namespaces[ns].metadata.labels["sandbox"]
  violation[x]
}

cc @timothyhinrichs for better ideas.

[timothyhinrichs]

Those conventions for writing policy make sense to me. I wonder more about
what OPA does with those policies.

Conceptually the admission control response only understands the strict
'deny'. To support severities of deny we'd need to think about what causes
OPA to evaluate the non-deny rules and what it does with those non-deny
results (e.g. feed to decision log or not).

Perhaps the simplest thing would be to have OPA config that specifies
additional decisions that should be run on every evaluation, the results of
which are included in the decision log. Any external actions those
additional decisions require could be handled via decision-log plugins.
I imagine it would be valuable for the list of additional decisions to be
part of policy, so that each policy could specify different additional
decisions.

Sketch....

package k8s.admission

normal k8s policy but with different severities

main = { ... deny ... }
deny[x] { violation1[x] }
alert[x] { violation2[x] }
ignore[x] { violation3[x] }

reserved name 'log' tells OPA what to log on every query (in addition to

the query)

Not sure about the syntax here. The below isn't clear enough; maybe

pragmas?
log = {deny, alert}

On Fri, Apr 26, 2019 at 7:39 AM Torin Sandall [e-mail]
wrote:

@phenixblue https://github.com/phenixblue thanks for filing this. If I
understand correctly, this is about controlling how the policy decisions
are handled at the enforcement point, e.g., does the resource get denied?
does it just trigger an alert? does it do both?

Today most admission policies are written as "deny" rules in OPA. However,
"deny" is not special in that it's just a name in the policy. We could
change the name to "violation". As long as we update the policy (or the
caller if they're explicitly querying "deny") the result would be the same.

violation["container missing requested memory"] {
containers[c]
not c.resources.requests.memory
}

Now the question is how to deny and/or alert when there's a violation.

To do this we could have rules like deny and alert.

deny[x] {
violation[x]
}

alert[x] {
violation[x]
}

The controller would query deny to decide whether to block the resource
and it would query alert to decide whether to alert on the resource.

With this convention we could fine-tune deny-vs-alert based on environment.

Deny resources with violations in the production namespace.

deny[x] {
input.request.namespace = "production"
violation[x]
}

Alert on resources with violations in non-"sandbox" namespaces.

alert[x] {
ns := input.request.namespace
not data.kubernetes.namespaces[ns].metadata.labels["sandbox"]
violation[x]
}

cc @timothyhinrichs https://github.com/timothyhinrichs for better ideas.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/open-policy-agent/opa/issues/1368#issuecomment-487081185,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACMVQKYELJJO3LZBQJGRMW3PSMHZVANCNFSM4HHUJVEA
.

[tsandall]

In the k8s admission control use case, we might get away with sticking the alerts into the main decision because I believe Kubernetes will be tolerant of additional JSON fields in the response (need to check this...) We could argue that we simply following Postel's law.

:thought_balloon: some prior art here might also be advice and obligations from XACML.

[timothyhinrichs]

That would be handy and would make it easy for a plug-in decision logger to
do whatever it needs with those decisions.

On Fri, Apr 26, 2019 at 1:22 PM Torin Sandall [e-mail]
wrote:

In the k8s admission control use case, we might get away with sticking the
alerts into the main decision because I believe Kubernetes will be
tolerant of additional JSON fields in the response (need to check this...)
We could argue that we simply following Postel's law.

💭 some prior art here might also be advice and obligations from XACML.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/open-policy-agent/opa/issues/1368#issuecomment-487189391,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACMVQK2ARVTBAWSIR77U6R3PSNQBXANCNFSM4HHUJVEA
.

[phenixblue]

@phenixblue thanks for filing this. If I understand correctly, this is about controlling how the policy decisions are handled at the enforcement point, e.g., does the resource get denied? does it just trigger an alert? does it do both?

Today most admission policies are written as "deny" rules in OPA. However, "deny" is not special in that it's just a name in the policy. We could change the name to "violation". As long as we update the policy (or the caller if they're explicitly querying "deny") the result would be the same.

violation["container missing requested memory"] {
  containers[c]
  not c.resources.requests.memory
}

Now the question is how to deny and/or alert when there's a violation.

To do this we could have rules like deny and alert.

deny[x] {
   violation[x]
}

alert[x] {
   violation[x]
}

The controller would query deny to decide whether to block the resource and it would query alert to decide whether to alert on the resource.

With this convention we could fine-tune deny-vs-alert based on environment.

# Deny resources with violations in the production namespace.
deny[x] {
  input.request.namespace = "production"
  violation[x]
}

# Alert on resources with violations in non-"sandbox" namespaces.
alert[x] {
  ns := input.request.namespace
  not data.kubernetes.namespaces[ns].metadata.labels["sandbox"]
  violation[x]
}

cc @timothyhinrichs for better ideas.

@tsandall Yes, your does the resource get denied? does it just trigger an alert? does it do both? comment is spot on.

For out internal tooling I basically wrote it to take the response message from each policy assessment and aggregate them into one statement that becomes the actual admission response and then keyed off that message to determine if an alert should be sent, and used the fixed DENY Level hierarchy to determine if the admission response should be allowed or denied.

I can confirm the Kubernetes API Server seems to tolerate additional fields being passed in the admission response beyond what is expected.

[abessifi]

Hi guys,

Is this issue still in design phase ?

I'm just wondering why not just to export violation events as a prometheus metrics and then let users leverage prometheus/alertmanager to define alerts in a more flexible way ?

What do you think ?

[patrick-east]

Is this issue still in design phase ?

Yes, I believe so.

I'm just wondering why not just to export violation events as a prometheus metrics and then let users leverage prometheus/alertmanager to define alerts in a more flexible way ?

With regards to the kubernetes admission controller use-case that is an ongoing discussion for Gatekeeper, check out https://github.com/open-policy-agent/gatekeeper/pull/657 for some more context on the current state over there.

For the kube-mgmt integration there are two services involved, the OPA server and the kube-mgmt instance. The OPA server exposes prometheus metrics, but they are generic (ie, OPA doesn't know anything about kubernetes, admission control stuff, what a violation is, etc). The kube-mgmt server is out of the loop on the actual decisions, so it would be difficult for it to report any either.

This is where the earlier suggestions of using the decision log for alerting comes in. You could write a decision log server (just a http server endpoint https://www.openpolicyagent.org/docs/latest/management/#decision-logs), configure OPA to send decision to it, and have the decision log service expose those prometheus metrics you were interested in. The logging service would need to understand the semantics of your policy result (eg, if you have a field "alerts" it needs look for that, parse them out, and generate the appropriate metrics). That contract however would be between the policies you write and the logger/monitor service implementation.

Going a little bit further, you can write a custom logger plugin in OPA https://www.openpolicyagent.org/docs/latest/extensions/#custom-plugins-for-opa-runtime which can even expose custom metrics. The benefit being that you keep the single OPA server instance, and reduce any overhead of running that second one. The downside is that until https://github.com/open-policy-agent/opa/issues/2348 is implemented you cannot add custom metrics to the OPA prometheus endpoint, so you would need to start another HTTP server for your plugin anyway.