Opa: Create rego code-documentation support to support generating documentation about policies

[petemounce]

https://openpolicyagent.slack.com/archives/C1H19LW4F/p1583922893200700

I have a policy like

# Every kind should have the app label
deny[msg] {
    not common.has_field(input.metadata.labels, "app")
    msg = sprintf("%s[%s] missing 'app' label. Has: %s", [input.kind, name, input.metadata.labels])
}

and I'd like to be able to extract documentation from the rego code (or, alternatively, associated closely _with_ it, say in a next-door file) and generate documentation for publishing. That to me means extracting the comments into markdown, then transforming that into static site generator of choice.

I'd want to be able to:

• give a policy a stable unique ID, so I can deep-link/index into my docs site easily (shellcheck does this very well, example: https://github.com/koalaman/shellcheck/wiki/SC1000).
• have support for defining the why of a policy in prose.
• have support for defining what "good" might look like, with ability to have a code snippet block.
• ditto, for defining examples of what "bad" might look like (although I suppose that's the policy rego itself).
• ask within a code review for documentation to accompany changes to regos (regi?).

I think this would also enable others to share policies more readily.

[itaysk]

Hi, I also have interest in this. We are starting to add a header to our policies with metadata about the policy. For us this is about documentation but also about integration into other tools (the system that uses those rego files).
We ended up defining something similar to a yaml front matter. We added it as a header to every file in a comment block. Formalizing this would be nice.
If this ever gets implemented, please consider allowing custom fields.

[itaysk]

as for good/bad examples - I think that a more appropriate place for those are in the policy tests. I personally liked how it's done in golang. Perhaps we could extent the test facility with "examples" semantics as well.

[tsandall]

We chatted about this on the community call yesterday. @johnharris85 did a neat demo of a tool they're building around OPA that implements some of these ideas.

One topic of discussion was how metadata should be defined. @jaspervdj-luminal pointed out that you can define the metadata directly in Rego. For example:

deny.rego (original rule)

package kubernetes.metadata.requireapplabel

deny[msg] {
    not common.has_field(input.metadata.labels, "app")
    msg = sprintf("%s[%s] missing 'app' label. Has: %s", [input.kind, name, input.metadata.labels])
}

metadata.rego (same package, different file, contains metadata about the policy)

package kubernetes.metadata.labelrequirements


title := "Label Requirements"

description := `The Label Requirements policy ensures
that all resources include an 'app' label for tracking
purposes`

tags := {"code": "SC1000"}

severity := data.policy.constants.CRITICAL

examples.rego (again, another file that contains example data)

package kubernetes.metadata.labelrequirements

example_good := {"metadata": {"labels": {"app": "nginx"}}}

example_bad := {"metadata": {"labels": {"owner": "bob"}}}

The package provides a namespace for the policy. Here we've got a 3-level namespace (domain=kubernetes, subdomain=metadata, policy=requireapplabel). Metadata can be defined in the same file or another. You can use raw strings for multi-line descriptions. Markdown could be embedded in those strings. Examples would require some convention (similar to tests like in Go).

A small library could be written on top of the github.com/open-policy-agent/opa/rego package to extract the metadata, examples, etc.

[johnharris85]

I've been playing with migrating from the approached demo'd (#-style doc strings) to this rego-native approach, although the multi-line string support is a bit limiting in terms of what Markdown is possible. For example code highlighting / fences are out because ` is used as the raw-string delimeter so something like the following errors:

description := `blah blah
    ```
    code fence
    ```
    blah blah

    blah blah.`

Could put the whole thing inside a single string with the appropriate newline characters and spaces but that ends up looking pretty ugly. Alternatively I could come up with some custom Markdown hack where ` is replaced with some other character and then flipped back later when I need to render it, but that feels a bit hacky and un-intuitive to the user. Anyone have any thoughts / ideas here?

Edit: Looks like ~~~ has decent support as a code fence, so that solves that. In-line code designation is still problematic though (as standalone ~ is ~strikethrough~):

`code`

[tsandall]

@johnharris85 I definitely agree that custom Markdown seems like a bad idea. Using a regular string w/ newline characters will be horrendous. A relatively simple solution (to implement) would be a new syntax for multi-line strings. Some thought would need to go into syntax (need to be careful not to overlap w/ anything else) and whether it's just an alternative to backtick (i.e., it generates a raw string) or whether there's anything else that should be done.

[itaysk]

In the last community call we discussed this issue and I drafted a proposal for your review. Appreciate any feedback as comments on the doc: https://hackmd.io/@ZtQnh19kS26YiNlJLqKJnw/H1gAv5nBw

[patrick-east]

Thanks for putting that together @itaysk 👍

One thing I really like about it is that it seems like that could pretty easily be done without any changes to OPA, as a purely convention-based thing + tooling that supports it, as a way to get things started.

[itaysk]

Thanks @patrick-east that was my aim. I'm hoping to discuss this in the next community call