Sensu-go: Feature Request: Allow arbitrary JSON data in ObjectMeta

[roganartu]

Sensu 1 allowed users to store arbitrarily deep JSON metadata on entities and checks. Sensu Go removes this in favour of a shallow map[string]string, forcing users to flatten data and lose the ability to store eg lists.

Expected Behavior

The following metadata structure should be supported.

{
    "entity": {
        "metadata": {
            "labels? annotations? attributes?": {
                "foo": {
                    "bar": ["baz"],
                    "baz": 2
                    "nested": {
                        "should": "work"
                    }
                }
            }
        }
    }
}

I don't mind where in the ObjectMeta it lives, and understand that it probably doesn't make sense to live under labels, and maybe also not under annotations. Perhaps attributes makes sense given that's effectively what they are.

Current Behavior

The only two key/val stores available to users in ObjectMeta are labels and annotations, and both only support string values.

If users are populating Sensu Go with existing nested attributes they need to flatten them:

{
    "labels": {
        "foo.bar": "baz"
    }
}

This doesn't work for lists though, which I have no workaround for.

Possible Solution

Add an Attributes field to ObjectMeta that takes arbitrary JSON and is exposed to the REST API. This would add this rich nested JSON support to everything that already leverages this type, including entities, checks, and events.

Context

2528

2674

[palourde]

Because labels are directly used by Sensu Go via multiple features (filtering, token substitution etc.) and because of Go relatively strong typing system, supporting custom structures in this object would add a massive performance overhead because it would require Go need to perform reflection at runtime. We actually did something like this in the earlier versions of Sensu Go and it was simply not sustainable.

On the other side, since annotations are not directly manipulated by Sensu Go, we could theoretically allow custom structures in there. The challenge we currently face is that while Go offers something like map[string]interface{}, there's no direct equivalence in Protocol Buffer (Protobuf), which is used for serialization in storage. That being said, we could possibly try something like https://stackoverflow.com/a/40265197 but we would have to make sure it doesn't impact performance.

[roganartu]

I could be mistaken, but last I checked annotations were only settable via command line flags on a sensu agent. Is that still the case? If so it would be helpful to also expose annotations to proxy entity creation flows.

[palourde]

Annotations and labels can be configured via sensuctl create command but we could easily add flags to the sensuctl entity create command unless you had something else in mind!

[roganartu]

Managing annotations via sensuctl create is perfect, I must've been mistaken thanks

[ninjasloth22]

Due to the way labels are currently tied in, and also with Annotations being rendered in the UI, I would suggest perhaps a new top level metadata attribute outside the scope of annotations and labels.
yaml example because i can't be bothered writing json...

metadata:
  annotations:
    foo: bar
  labels:
    environment: production
  custom:
    freeformcustomdata:
       list: ["item1", "item2"]
       nestedhash:
          foo: bar

etc.
This is coming from a stand point of not reading through the code, however if we were to pass in quite freeform data, this avoids the label system and prevents you having to worry about message data structures being rendered in the UI as annotations currently are.
It would really just be data that can be sent with the event for processing via handlers etc.

[roganartu]

It would really just be data that can be sent with the event for processing via handlers etc.

For context this is basically the use case that I raised the issue for; to access arbitrary data stored in entities during event processing without having to lookup that data externally. I don't mind if this only gets implemented specifically for entities instead of all resources via ObjectMeta if that's less technically problematic.

Sensu is not the source of truth for our entities. We have a script that periodically syncs entities from our source of truth to Sensu. This source of truth contains rich entity details in the form of arbitrary JSON, much of which would be useful when handling events but would be costly to lookup by querying the source of truth from inside a handler. Making this arbitrary data available inside the event pipeline would make intelligently handling events significantly easier.

[ninjasloth22]

This is exactly our use case as well. We sync many of our entities in from external sources.
However for our case we would love to have this option on both entities and checks, as we do use this metadata for routing logic within our custom handlers.

[palourde]

This is basically the intent we had with annotations; allow arbitrary data to be passed along the pipeline. Sensu itself wouldn't manipulate those, but handlers could do whatever they want with that information.

with Annotations being rendered in the UI

@ninjasloth22 I'm not sure to understand that statement, do you mind clarifying a bit?

Thanks!

[ninjasloth22]

I was just referencing the fact that Annotations are currently rendered in the UI.
With regards to adding in a more free form data structure I was just suggesting if it meant less code work on your end then a different metadata attribute could also be considered that is not rendered in the UI(rather than using annotations). As per my example above.
However I am totally for using annotations it was just a "if it means more work to update/use annotations - something else could be considered"....if that makes sense :)

[calebhailey]

Because labels are directly used by Sensu Go via multiple features (filtering, token substitution etc.) and because of Go relatively strong typing system, supporting custom structures in this object would add a massive performance overhead because it would require Go need to perform reflection at runtime. We actually did something like this in the earlier versions of Sensu Go and it was simply not sustainable.

On the other side, since annotations are not directly manipulated by Sensu Go, we could theoretically allow custom structures in there. The challenge we currently face is that while Go offers something like map[string]interface{}, there's no direct equivalence in Protocol Buffer (Protobuf), which is used for serialization in storage. That being said, we could possibly try something like https://stackoverflow.com/a/40265197 but we would have to make sure it doesn't impact performance.

This is correct. The simple way to think about labels vs annotations is: labels are for selectors, annotations are for data.

Due to the way labels are currently tied in, and also with Annotations being rendered in the UI, I would suggest perhaps a new top level metadata attribute outside the scope of annotations and labels.

Adding another attribute is just moving the problem IMO. I have questioned whether it's appropriate to render annotation data in the UI. It's available from the API and sensuctl, so removing it from the UI wouldn't make it inaccessible for power users.

@roganartu @ninjasloth22: I have had great success with using annotations to store arbitrary data as escaped JSON; it's still a string, and it's super easy to process. And from a "human readable" perspective, I have found that "embedding JSON in YAML" isn't as nasty in practice as it sounds when you say it out loud. For example:

type: CheckConfig
api_version: core/v2
metadata:
  name: example_check
  namespace: default
  annotations: 
    custom_data: |
      [
        {
          "foo": "bar",
          "baz": [ 1, 2 ]
        }
      ]
spec:
  command: example_check.sh
  publish: true
  interval: 10
  subscriptions:
  - test
  timeout: 10
  handlers:
  - slack

I'd love to get your thoughts on this approach. Have you tried it? Where does it break down for you?

[ninjasloth22]

Interesting suggestion! I have not tried it let me get back to you after some testing

[ninjasloth22]

This does seem to be a workable solution. However it does raise issues for us around the self service aspect and having to validate and clean the json data coming in as a string. Things we can obviously work around but it would be good to have a more solid solution going forward.
P.S large json strings look like S* in the UI when they are rendered with their new line and escape characters :)

[calebhailey]

Do you even need to see annotation data in the Web UI? What if we only displayed labels? Use labels for simple attributes, stringified JSON for more complex data structures.

Thoughts?

[ninjasloth22]

Personally I have no need to see Annotations in the webUI.
However it would just depend if other users are attempting to render documentation links etc with them?
Though they could also move those labels if needed.

[echlebek]

Closing this due to inactivity, but perhaps this discussion should be represented in our Discourse forum!