Sensu-go: Stash API

[portertech]

Sensu 1.x provides a stash HTTP API, which is essentially a JSON blob store via HTTP.

1.x docs https://sensuapp.org/docs/1.1/api/stashes-api.html#sensu-stashes-api

Implement a stash API in the Backend API.

[grepory]

If people want to use stashes, we can revisit this later.

[runningman84]

Stashes are quite important for us. We cache information used by handlers there. For example aws sts tokens and other kind of expensive api calls. This allows us to run any amount of handlers in the most efficient way.

[apognu]

Same here, some of our checks consist of comparing previous data with new one for alerting. Any chance the Stash API would come back?

[palourde]

We do not have any plan to bring stashes to Sensu Go in a near future.

Labels and annotations can now provide additional context to handlers and other resources, while the Sensu Go API can be easily queried by handlers.

We are definitely interested in hearing more about your use cases if you do not believe these alternatives are viable for you.

Thanks!

[apognu]

My use-case is having an (external) check for SSL certificates found in CT logs. I want to throw an alert each time a new certificate is issued for my domain. As far as I can tell, this would require retrieving the results from the previous check and alerting if a new entry is found in the current check.

I do not need historical data (Sensu would not be the right choice for that) but simply have some kind of state between a check and the previous one.

[palourde]

Here's an idea, let me know if you think it would work. Could the agent running this check caches the state and keep track of the certificates? Instead of having all this logic living in a handler, the check itself could determine if a new certificate was issued.

[apognu]

This is actually what I had in mind. My thought was the check running on the agent would access the Stash API to compare new state against previous state, and return a warning whenever a certificate is issued.

My agent runs on Kubernetes, so storing state on the host's filesystem would be a no go without pinning the agent on a specific node.

I do have a workaround by storing state on external blob storage, but it would be more convenient if everything was handled by Sensu.

I do understand perfectly if there is no intention of reinstating this API, though.

[adammcdonagh]

Adding my use case for Stashes here...
We use custom SNMP traps for forwarding Sensu alerts into our centralised alerts view from Sensu.
Many of our checks monitor multiple things.. e.g. for filesystems, the check result contains 1 line per filesystem, along with it's state, severity of the SNMP trap that should be generated.

In this case, because multiple things are monitored by the check, we need a way to store which specific line in the check generated the error, and store the previous state... This way, if an alert severity changes from warning to critical, we can check in the stash, see that we previously sent out a warning alert, send a clear for the warning, and then send a new SNMP trap for the critical alert.

In addition to this, if 1 filesystem changes from good to bad, and another changes from bad to good at the same time, Sensu would not see a change in state, so our handler needs to process the WARN or CRIT lines and determine (based on what's stored in the stash) which combination of SNMP traps to send.

We did try doing this on the agents, but it then meant that every check of this kind would need to have some kind of local cache, and working out what results to return became overly complex...

Stashes were perfect, because they were centralised, and no matter which Sensu node picked up the check result, it could determine the appropriate action to take...

Perhaps our only option at the moment is to use something like Consul and store K/V pairs in there, unless we can access the internal etcd directly (I'd rather not have to manually configure my own etcd cluster).

[calebhailey]

+1! It will be super interesting to revisit this following the release of Secrets Management! 🤞

[nikkictl]

@echlebek wants to implement Redis! We probably need a proposal for this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.