Kibana: 403 when user enables Monitoring with sufficient roles

[igoristic]

Kibana version: 7.x - 8.0.0 (master)
Elasticsearch version: 7.x - 8.0.0 (master)

When a newly created user with sufficient (built-in) monitoring roles: monitoring_user and kibana_user (source: https://www.elastic.co/guide/en/kibana/current/monitoring-data.html) clicks on Stack Monitoring from the side-menu they are presented with two 403 errors:

[cluster:monitor/state] is unauthorized for user [test]
[cluster:monitor/nodes/info] is unauthorized for user [test]

I created a new role monitoring_new with the following cluster privileges:
monitor - to get rid of the two initial 403 errors (mentioned above)
cluster:admin/settings/update - to get rid of an additional 403 error (caused by modifying xpack.monitoring.collection.enabled) when clicked on [Turn on monitoring]

This should probably be fixed on https://github.com/elastic/elasticsearch side, since the roles are derived from: GET /_security/role

[elasticmachine]

Pinging @elastic/stack-monitoring

[elasticmachine]

Pinging @elastic/kibana-security

[ycombinator]

I was able to reproduce this yesterday running the latest master versions of Kibana and Elasticsearch. Basic license, security enabled.

[ycombinator]

This should probably be fixed on https://github.com/elastic/elasticsearch side, since the roles are derived from: GET /_security/role

This may very well be the case, but it would be good to understand a few things first IMO:

1. Why is this breaking now? What changed recently? Does this have anything to do with moving Security features to Basic (which is a recent change)? Does this still break when using a Trial license instead of Basic?

2. We know this breaks with Kibana master against Elasticsearch master. What about other recent version combinations like Kibana 7.x and Elasticsearch 7.x? Can we isolate if this is a breakage in a particular version of Kibana or a particular version of ES?

3. Finally, specifically what code paths in the Monitoring Kibana code are making ES API calls that require the privileges listed in the PR description?

[cachedout]

To the excellent points already raised by @ycombinator , I would add that if this is has recently broken, what test coverage are we missing which would have detected this?

[chrisronline]

I tested this on master and reproduced this issue, but through more testing, discovered there was a scenario where this would not happen.

If you start from a fresh master, enable monitoring (through the UI by clicking the Turn on monitoring button) using the super user (by default, elastic user) first. After doing this, the link for the monitoring user (with kibana_user and monitoring_user) will show the monitoring UI as expected.

Pinging @ycombinator and @igoristic to verify/reproduce

Assuming so, this might affect the severity of this issue, to some degree. I'm not sure what is more common for users - to create a specific monitoring user _before_ monitoring is enabled, or _after_.

[ycombinator]

I was able to successfully reproduce the scenario @chrisronline described above. Nice find!

Also, I tested this same scenario with 7.0.1 (ES and Kibana) and I was able to reproduce it there as well. So I think it's reasonably safe to say that no recent changes have caused this issue.

Assuming so, this might affect the severity of this issue, to some degree. I'm not sure what is more common for users - to create a specific monitoring user before monitoring is enabled, or after.

Given that this doesn't seem to be a result of a recent change (i.e. this is not a regression) plus the fact that this hasn't cropped up in the past, I tend to agree that the severity might not be as high any more.

That said, I do think this is an opportunity to improve the UX. To that end, I think we have a couple of options:

1. We expand the privileges granted by the built-in monitoring_user role. Before we do this, IMO we should first understand exactly what ES API calls the Monitoring code is making to cause these errors.

2. If, for some reason (likely related to security), we decide not to do (1), we should then:

   1. Document this pre-requisite of enabling collection using a user with the superuser role, and
   2. See if we can detect this situation in the Monitoring UI and make a similar suggestion as the documentation in the UI itself.

Thoughts?

[chrisronline]

we should first understand exactly what ES API calls the Monitoring code is making to cause these errors.

When there is no .monitoring-* data found, the UI will attempt to detect the reason _why_. This involves a few checks, notably a check against cluster settings and a check against node settings. These two checks are the failing checks resulting in the errors we're seeing in the UI for this kibana_user and monitoring_user user.

Of the two checks, one seems drastically more important than the other. I can't find anywhere in our code that calls /_nodes/settings again, so I'm not sure how to handle that.

We do need the _cluster/settings endpoint when we eventually enable monitoring (enabling monitoring through the UI just sets the xpack.monitoring.collection.enabled cluster setting to true).

If I try and hit this endpoint using the kibana_user and monitoring_user user, I see a security error:

action [cluster:admin/settings/update] is unauthorized for user [monitoring]

AFAIK, there is no other write action happening within monitoring other than to the _cluster/settings endpoint.

Therefore, I think if we go with option 1 (which I think we should), we should extend the monitoring_user role to also include the cluster:admin/settings/update privilege, which I'm pretty sure is controlled here

[igoristic]

I was also able to reproduce @chrisronline's find (nice one!). This does lessen the severity a bit, and probably why it's been dormant for so long

I'm also in favor of option 1 since then:

1. We're not adding an extra step
2. We don't have to change the documentation

[chrisronline]

I just verified that adding the cluster:admin/settings/update permission to the default monitoring_user role allows the kibana_user and monitoring_user user to successfully execute the PUT _cluster/settings call, necessary to enable monitoring.

However, the solution we want it's slightly more complicated.

If we only update the built-in monitoring_user role to have the cluster:admin/settings/update, the user will still see the same errors in the UI. Maybe we can just change the UI to not make those same checks, but those checks are valuable (except perhaps the /_nodes/settings one) and we still want to perform them to help identify why the user is not seeing monitoring data.

In order to make the above check work properly, we also need to give the built-in monitoring_user role, the cluster:monitor/state permission, which allows for the reading of cluster settings.

With both of these new privileges added, I think we'll achieve the desired UX.

[igoristic]

To the excellent points already raised by @ycombinator , I would add that if this is has recently broken, what test coverage are we missing which would have detected this?

Most (if not all) of our tests are executed by the default Kibana superuser. I think with a little bit of fiddling we can execute those same tests as a "created" user with: kibana_user and monitoring_user roles and just assume everything will work the same for the default superuser (so we don't have to run them from both points of view)

[chrisronline]

@igoristic @cachedout We have a ticket for this already, btw: https://github.com/elastic/kibana/issues/29661

[ycombinator]

I agree option 1 is the better UX but I think it's worth bringing up a few other considerations:

1. Currently the built-in monitoring_user role only has read access to the cluster. By adding the cluster:admin/settings/update privilege to this role we would be also giving users write access to cluster settings. And AFAIK there's no way to further limit the scope of such access, i.e. only allow updates to _monitoring_ cluster settings. So we need to think about what other potential harm a user could cause to their production cluster.

   BTW, expanding the role to include just the cluster:monitor/state privilege is relatively okay as that is still a read-only operation.

2. The xpack.monitoring.collection.enabled setting controls whether _internally collected_ monitoring data by Kibana, Logstash, Beats will be accepted by the ES production cluster for enriching and exporting to the monitoring cluster. As such, in the longer term with Metricbeat collection, users will not need to rely on this setting at all.

3. Finally, as we've learnt, this situation has existed at least as far back as 7.0.1. AFAICR we haven't seen any feedback from users about this.

In summary, there's clearly a benefit from option 1 but there's a cost (reduced security) as well. And given the long-term fate of the setting in question + lack of any bug reports around this situation, I wonder if the benefit is really worth the cost.

---

I wonder if there is a relationship here to the "Setup mode" that we're introducing in the near future. I wonder if we could say that there are certain privileged operations that can only be performed in Setup Mode and using that mode requires elevated access. Setting up monitoring (for internal collection) would be one such action. Similarly, turning it off (to setup external collection) would be another.

[cachedout]

I don't really like the first option. Though it's much cleaner from a UX perspective, giving a monitoring user write access to cluster settings is a very strong anti-pattern, IMHO. I suspect that we would experience significant pushback from our users if we were to do this.

I would rather the setup process require a few extra steps than leave our users with a monitoring user with permissions which are potentially destructive.

[chrisronline]

I wonder if there is a relationship here to the "Setup mode" that we're introducing in the near future. I wonder if we could say that there are certain privileged operations that can only be performed in Setup Mode and using that mode requires elevated access. Setting up monitoring (for internal collection) would be one such action. Similarly, turning it off (to setup external collection) would be another.

++. Definitely something we could do with this

[cachedout]

See related discussion here: https://github.com/elastic/kibana/issues/38576

[chrisronline]

We resolved this mostly with https://github.com/elastic/kibana/pull/50421 where the user will see a message telling them they do not have the required permissions to setup monitoring.