Azure-rest-api-specs: Redis API is masking the account key being used

[tombuildsstuff]

:wave:

We're seeing an issue with the Redis API where the RDB Storage Connection string sent to Azure is different to the one returned from the API, for instance we see the response:

DefaultEndpointsProtocol=https;BlobEndpoint=https://unlikely23exst2acct4u0j.blob.core.windows.net/;AccountName=unlikely23exst2acct4u0j;AccountKey=[key hidden]

rather than the original value posted

DefaultEndpointsProtocol=https;BlobEndpoint=https://unlikely23exst2acct4u0j.blob.core.windows.net/;AccountName=unlikely23exst2acct4u0j;AccountKey=Fv5Ien72PEKr++K8tp5L/fR3qlPghHUiRqHZZS25dR9LsnD//tz8lXLGRORCkBnwrXxltCrx2qjEOZAmGIEgGA==

which in our case is leading to spurious diff's; whilst we could look into ignoring the returned "[key hidden]" value, this makes it impossible for us to detect changes.

Would it be possible to fix the API to return the key that's being used?

Thanks!

[TimLovellSmith]

@tombuildsstuff BTW don't forget to rotate that storage account key!

[tombuildsstuff]

@TimLovellSmith should have mentioned this was an acceptance test, so the Redis Cache in question's already been deleted :)

[TimLovellSmith]

@tombuildsstuff
We are intentionally masking it out in responses to help you avoid unintended information disclosure, since azure permissions to view related storage account credentials should not be implied by azure permissions to get general information about a redis cache.

[tombuildsstuff]

@TimLovellSmith so how are users expected to know which access key is being used when rotating secrets for example? Whilst this is potentially useful for scenarios in the Azure Portal; this approach has the potential to cause production incidents for example where an operator is unsure which key is currently in use, and cycles the wrong one. As such can we look into adding an override/flag to return this information?

Thanks!

[TimLovellSmith]

@tombuildsstuff
Assuming you have enough permissions and you don't know which key is being used, and you want to rotate specifically one of the keys, then you can just reconfigure your redis cache to use the key you are not going to rotate, before you rotate.

[tombuildsstuff]

@TimLovellSmith whilst that approach could work, that's not a suitable workaround in all situations unfortunately - other API's within Azure expose a sensitive endpoint which returns sensitive information; could we get a similar endpoint added to the Redis API?

cc @joshgav @marstr @jhendrixMSFT - this is the issue I was referring too in our meeting the other day

[marstr]

Adding labels to help with our tracking systems.

[TimLovellSmith]

@tombuildsstuff Got some examples of those APIs? I don't know what exactly you are thinking of and it would probably be useful to be able to make the same comparison.

[tombuildsstuff]

@TimLovellSmith off the top of my head - AKS returns an access profile which includes all the sensitive K8S information; App Service returns the connection strings etc.

[joshgav]

@tombuildsstuff what API(s) are you calling that return this obfuscated response? Thanks!

[joshgav]

Hi @tombuildsstuff is this still an issue or did you work around it in the end? Thanks!

[tombuildsstuff]

@joshgav sorry I've been AFK until today.

what API(s) are you calling that return this obfuscated response? Thanks!

In this case this is the Get API's redisConfiguration dictionary, which is masking the output.

is this still an issue or did you work around it in the end? Thanks!

This is still an issue for us at this time - since we're unable to know which Storage Access Key is being used by the Redis Cache.

Thanks!

[TimLovellSmith]

@tombuildsstuff you mentioned way back at the start of the thread that you could look into ignoring the returned "[key hidden]" value, could you still do that? I think this is still doings its intended behavior, since these storage connection strings if exposed may effectively leak the data which is stored in the cache.

[tombuildsstuff]

you mentioned way back at the start of the thread that you could look into ignoring the returned "[key hidden]" value, could you still do that? I think this is still doings its intended behavior, since these storage connection strings if exposed may effectively leak the data which is stored in the cache.

@TimLovellSmith taking that approach that means we'd be unable to detect changes to the Storage Account Key, which means that Terraform's Diff would work inconsistently across the Redis Coiguration; as such I don't think that's an appropriate workaround in this instance unfortunately.

Since the Redis Configuration can contain sensitive values, is there a reason this couldn't be moved out to a sensitive endpoint ala GetAccessProfile in AKS or ListKeys in the Storage Accounts?

Thanks!

[TimLovellSmith]

@tombuildsstuff I don't know Terraform's Diff - what is it supposed to do?

[tombuildsstuff]

@TimLovellSmith sorry missed this.

@tombuildsstuff I don't know Terraform's Diff - what is it supposed to do?

Terraform's Diff (or plan) functionality shows you the difference between the current state of the resource (for example, the Connection Strings used in Azure) and the state defined locally in code - and shows you a proposed diff. The App Service API as an example returns the full (unmasked) connection string, so it's possible for users to see which parts of the connection string have changed in the diff.

Hope that helps :)

[TimLovellSmith]

@tombuildsstuff
So basically this behavior matters only to people who are storing their connection strings in their code?

[TimLovellSmith]

@bsiegel Can we close this? This behavior is by design, to avoid accidentally leaking sensitive connection string data.

[tombuildsstuff]

@TimLovellSmith

Can we close this? This behavior is by design, to avoid accidentally leaking sensitive connection string data.

More modern API's (Storage, AKS, Media Services etc) follow a different convention where these keys are exposed on a sensitive privileged endpoint - as such there's precedent for the Redis API to follow here IMO - as such I believe this bug report/feature request is still valid?

[TimLovellSmith]

@tombuildsstuff What pattern are you referring to - keyvault scenarios? Or something else?

[tombuildsstuff]

cc @vladimirjoanovic

[iitsDelbruegger]

I am deeply disappointed that this bug is not fixed yet. We are setting up all our infrastructure in Terraform and automatically store the secrets in a key vault so that no developer or admin has to fiddle around with them - but Redis cannot output the connection string because of this bug. Do you not want to support modern Infrastructure as Code solutions?

[TimLovellSmith]

@iitsDelbruegger We can't really solve this problem in the ways suggested without also having security impact. I think the ideal solution here is a non-key-based approach for configuring redis authentication to storage, such as System Assigned Identity, which could both eliminate problems with storage key rotation breaking persistence access, and mitigate the possible problems with exposing credentials, and terraform integration.

[tombuildsstuff]

@TimLovellSmith as mentioned above, the AKS API provides one example for doing just that: https://docs.microsoft.com/en-us/rest/api/aks/managedclusters/listclusteradmincredentials - which gives you full access to an AKS Cluster including (via provisioning pods etc) any resources it has access too, which I'd argue is more sensitive.

That said, I'd agree it could make more sense for the Redis RP to use a Managed Identity to look this information up going forward - but that's a whole separate solution to the problem.

From Terraform's side, just knowing if it's "primary" or "secondary" (or "no longer valid", if the keys been rolled, but the redis cache hasn't been updated) could be a sufficient interim fix too rather than returning the entire key otherwise (presuming the API goes through and raises an issue should this connection string no longer work to be able to detect the "no longer valid" state).

WDYT?