Elasticsearch: Remove API key name uniqueness constraint across active API keys

[bizybot]

When API keys were introduced they were primarily driven by Hadoop ES use case. Naming an API key was a nice to have feature but got preemptively built with constraints like it being unique across active API keys.

The current design does not guarantee this behavior as when we create an API key we do not refresh index immediately (the default is wait_for) and so it is not available when creating another API key with the same name where the request arrives before the refresh.

Given that we are now using API keys across the board for various use cases like Cloud, Hadoop, Fleet, Alerting, etc. we need to think whether we want to continue with this constraint.

As far as I remember the comment was to have a named API key but whether it should be unique across API keys was something that we decided runtime as I could not find/remember any discussion around it.
Although minor another possible problem to keep into consideration is the indexing/creation speed that gets impacted due to this constraint.

This issue exists to discuss the following:

• Do we think the constraint of unique API key name is of value?
• If not can we remove this constraint? so the name can be considered as a tag.
• If yes do we want to refresh immediately(as default) after indexing new API key document?

[elasticmachine]

Pinging @elastic/es-security

[bytebilly]

@bizybot if I get the context correctly, the uniqueness is about the name property of the API key (not the id). The following considerations are assuming this is correct.

From a technical point of view, having the name field unique is not a strict requirement, since we already have the id field that guarantees we can distinguish two keys with the same name.

In my opinion, the problem arise when keys are managed by the user. This is something we may expect to happen via APIs, and also via the Kibana UI (see https://github.com/elastic/elasticsearch-team/issues/228 for context).
In such context, users will primarily use the name field to recognize keys and their scope, rather than the id that is an internal opaque attribute that cannot be helpful to remind.

If we don't guarantee uniqueness of the name field, users can create two keys with the same value: not a good practice, but it may happen for a variety of reasons.
If they need to revoke one of the two, they will not know which is the right one.

That's why I think that having name unique is important. It should probably be unique per-user, so different users may have the same key name. This would require adding the "owner" in the equation, but with users being able to manage their own API keys in the next releases, it could make sense anyway.

If you look at GitHub PAT, for example, they enforce note (the equivalent of our name) to be unique.

What do you think?

[bizybot]

Thank you @bytebilly for your comment.

Your understanding is correct, the Id field is there for distinguishing two API keys and the purpose for name was to identify API key by some user-defined tag.

I agree with uniqueness amongst all API keys may not be desirable but per user makes sense.

If we want to guarantee this behavior then we need to refresh the index on indexing new API key document immediately. This will ensure that when we query for duplicates we can see those.
Then this will be similar to the put user request where the refresh is immediate.

And where we require performance of API key creation (which so far has not been a concern) the users can set appropriate refresh policy in the Api key request.

[tvernum]

the users can set appropriate refresh policy in the Api key request.

But won't that mean they are able to bypass the uniqueness check?
Are we just saying the unqiueness check is best efforts?

[bytebilly]

the users can set appropriate refresh policy in the Api key request.

But won't that mean they are able to bypass the uniqueness check?
Are we just saying the unqiueness check is best efforts?

I probably misunderstood that. I thought it means they have to wait the refresh interval (whatever they set) before the API call returns.

I'd suggest to consider the uniqueness as a strict constraint, if we want to make it predictable and allow users to rely on it. This may impact the way we'll present keys in the UI too.
Is forcing the refresh a performance concern? I'd expect the API keys will not be created too frequently, but obviously I understand we want to consider the case.

[tvernum]

I thought it means they have to wait the refresh interval (whatever they set) before the API call returns.

If you want to optimise for performance you want to avoid any forced refresh, but neither do you want to wait for the standard refresh time.
There isn't really an option to strictly enforce uniqueness (by user or globally) and also optimise for bulk creation (particularly in the current API which has no bulk creation options).

[bizybot]

~If we want to guarantee this behavior then we need to refresh the index on indexing new API key~

+1 that there is no option to enforce strict uniqueness unless we add it to the id of the API key created which we may not want to do.

I am leaning towards not having this constraint of having a unique API key name and treat it as a tag/additional metadata.
A user might have the same tag for multiple keys and there is a creation time to differentiate between them. It may be used to tag API keys like service: finance, service: hr by admins creating API keys for different apps connecting to ES.
@bytebilly @tvernum thoughts?

[tvernum]

My vote would be to not enforce any uniqueness in the API. We can't do it well, and I don't think we should do it badly.

There's also no real reason to enforce it - it doesn't help us, and if they're system generated keys (e.g. fleet) then what do we expect the system to do? Just keep appending randomness until it works?

The UI could use the GET API with name and owner=true to check for an existing key before saving, if we think that would be a usability improvement, but I don't think it belongs in the API.

[bytebilly]

I'm still on my idea that uniqueness could help to improve UX of API keys.

However, since there is no simple and clean way to reach this goal via the API, and it is not a strict constraint, I agree it could be done in the UI, if we agree it can bring benefits.

[bizybot]

Based on the above discussion, I will create a PR to remove the uniqueness constraint for the API key name as we cannot guarantee the uniqueness. Thank you.

[bytebilly]

@jethr0null @cjcenizal this is not strictly related to the first iteration we are working on for API key management UI in Kibana, but we should reconsider and discuss about that if/when we'll add the ability to create new entries in the UI.

[cjcenizal]

Thanks @bytebilly.

I'm still on my idea that uniqueness could help to improve UX of API keys.
However, since there is no simple and clean way to reach this goal via the API, and it is not a strict constraint, I agree it could be done in the UI, if we agree it can bring benefits.

I agree with this.

The UI could use the GET API with name and owner=true to check for an existing key before saving, if we think that would be a usability improvement, but I don't think it belongs in the API.

We can use this workaround that Tim suggested.

[tvernum]

We can use this workaround that Tim suggested.

@jkakavas raised a concern (in the security area meeting) that, if the UI prevents something, then users will make assumptions that it can never happen and might treat the name as unique even though it isn't.
I don't have a strong view, about whether that should change our approach (e.g. to warn the user rather than strictly prevent them from creating a duplicate), but I think it's worth capturing here.

[bytebilly]

I agree warning instead of blocking is a reasonable approach — we cannot prevent it from happening anyway since it will be possible via API.

Since this check is just for user's UX, if they know and they want to create it anyway... they will deal with it 🙂

[albertzaharovits]

Although people's names are not unique, the name qualifier does raise some expectation of uniqueness. I suggest we present the name as a tag in the UI. On the ES side we could also return a tag alongside a name which would be having identical values in 7.x but name would be removed in 8 (due to the lack of API versioning).

[bytebilly]

Since we agreed on the backend changes in the API, I created another issue to discuss the specific proposal for UI behavior and we can move the conversation there: https://github.com/elastic/elasticsearch/issues/47502