Aws-cdk: [aws-elasticsearch] Elastic search domain resource lacks permissions.
Elastic search domain resource lacks permissions.
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::<ID>:assumed-role/Testing-Mi-Main-Stack-<ID>-<ID>/Testing-Mi-Main-Stack-<ID>-<ID> is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-central-1:<ID>:key/<ID> (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: <ID>; Proxy: null)
Reproduction Steps
Create elastic search Domain. (from aws_cdk.aws_elasticsearch import Domain).
What did you expect to happen?
Domain created.
What actually happened?
CloudFormation error complaining about permissions:
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::<ID>:assumed-role/Testing-Mi-Main-Stack-<ID>-<ID>/Testing-Mi-Main-Stack-<ID>-<ID> is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-central-1:<ID>:key/<ID> (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: <ID>; Proxy: null)
Environment
- CDK CLI Version : 1.72.0
- Framework Version: 1.72.0
- Node.js Version: v15.1.0 -->
- OS : MacOS
- Language (Version): Python3.9
Other
The bug seems to be new. Did not experience in previous versions.
This is :bug: Bug Report
laimonassutkus
All 34 comments
Anyone?
laimonassutkus
on 11 Nov 2020
Same issue here. It worked before with the same configuration
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::<>:assumed-role/<>/<> is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-west-1:<>:key/<> (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: <>; Proxy: null)
Ruben-E
on 13 Nov 2020
I was using a custom KMS key. Just created the same domain but with the default key instead and that worked.
Deployed multiple cluster with the same setup (with the custom key) before, whats changed that it doesn't work anymore?
Ruben-E
on 13 Nov 2020
Cmon @iliapolo, help us :))
laimonassutkus
on 13 Nov 2020
@Ruben-E using default kms keys is a bad practice.
laimonassutkus
on 13 Nov 2020
Well, For us it's an acceptable workaround for now. At least able to proceed with testing.
And is the default key actually considered as bad practice? The custom key I used was also managed by AWS.
Ruben-E
on 13 Nov 2020
With a custom master key you can control who has access to it and who does not. Wit default one you can't do that. Also, with custom master keys you can enable key rotation.
P.S. @iliapolo, where are you?
laimonassutkus
on 14 Nov 2020
@iliapolo @eladb why no one is taking care of this critical bug? Guys? Hello?
laimonassutkus
on 23 Nov 2020
@laimonassutkus apologies for the delayed response. I will have a look at this today.
iliapolo
on 23 Nov 2020
Thanks! Appreciated.
laimonassutkus
on 23 Nov 2020
@laimonassutkus @Ruben-E Can you please share the code you are using exactly?
iliapolo
on 23 Nov 2020
@iliapolo A piece of the code:
const kmsKey = new Key(this, 'EncryptionKey', { enabled: true });
...
new CfnDomain(this, 'Elasticsearch', {
...
encryptionAtRestOptions: {
kmsKeyId: kmsKey.keyId
}
...
})
@Ruben-E I'm not able to reproduce this..we indeed didn't change anything in module for a while, especially not in the L1 resources.
This is the code i'm using:
const key = new kms.Key(this, 'DomainKey', { enabled: true })
new elastic.CfnDomain(this, 'Domain', {
elasticsearchVersion: '7.7',
encryptionAtRestOptions: {
kmsKeyId: key.keyId,
enabled: true,
},
ebsOptions: {
ebsEnabled: true,
volumeSize: 80
}
})
I wonder if just a problem with the role you are using when deploying. Im using an administrator role.
Can you share how you run cdk deploy? Are you explicitly assuming a role before running the command?
P.S just making sure you are aware that we have an L2 construct now: elasticsearch.Domain.
iliapolo
on 23 Nov 2020
@iliapolo
Thanks for checking it. 10 days ago I ran into this issue, maybe its working again now. The code did work before, maybe it was a temporary hiccup? I'll test it again later today.
I shared just a piece of our code, but replacing that custom kms key with the default key made it work again, so I think the example you shared is comparable with the situation and should fail as well.
I know about the L2 construct, thanks for the reminder :). But I doesn't / didn't support audit logs which was recently added. Switched to the cfn resource to support that. I have it on my list to create a PR for this ;)
Ruben-E
on 23 Nov 2020
@Ruben-E cool 👍
Regarding the error though, could it be that the role you are running doesn't have the necessary KMS permissions?
Could you also share the output of cdk deploy?
iliapolo
on 23 Nov 2020
Here's a reproduction case in Python
from aws_cdk.core import Stack, App, RemovalPolicy
from aws_cdk.aws_kms import Key
from aws_cdk.aws_iam import PolicyDocument, PolicyStatement, Effect, AccountRootPrincipal
from aws_cdk.aws_elasticsearch import (
AdvancedSecurityOptions,
CapacityConfig,
Domain,
EbsOptions,
ElasticsearchVersion,
EncryptionAtRestOptions,
TLSSecurityPolicy,
ZoneAwarenessConfig,
)
from aws_cdk.aws_ec2 import EbsDeviceVolumeType
class Infrastructure(Stack):
def __init__(self, scope: Stack):
super().__init__(
scope=scope,
id="EsReproStack",
stack_name="EsReproStack",
)
kms_key = Key(
scope=self,
id="EsReproCmk",
alias="EsReproCmk",
enabled=True,
enable_key_rotation=True,
policy=PolicyDocument(
statements=[
PolicyStatement(
actions=[
'kms:*',
],
effect=Effect.ALLOW,
resources=['*'],
principals=[AccountRootPrincipal()]
),
]
),
removal_policy=RemovalPolicy.DESTROY,
)
domain = Domain(
scope=self,
id="EsReproDomain",
version=ElasticsearchVersion.V7_7,
access_policies=[
PolicyStatement(
actions=["es:*"],
effect=Effect.ALLOW,
resources=["*"],
principals=[AccountRootPrincipal()],
),
],
capacity=CapacityConfig(
data_node_instance_type="t3.small.elasticsearch",
data_nodes=1,
master_nodes=None,
),
ebs=EbsOptions(enabled=True, volume_size=10, volume_type=EbsDeviceVolumeType.GP2),
enforce_https=True,
encryption_at_rest=EncryptionAtRestOptions(
enabled=True,
kms_key=kms_key,
),
node_to_node_encryption=True,
tls_security_policy=TLSSecurityPolicy.TLS_1_2,
use_unsigned_basic_auth=False,
zone_awareness=ZoneAwarenessConfig(enabled=False),
)
app = App()
Infrastructure(app)
app.synth()
Relevant parts of output from cdk deploy
3:42:24 PM | CREATE_FAILED | Custom::ElasticsearchAccessPolicy | EsReproDomainESAccessPolicyDCE7667B
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::485225193788:assumed-role/EsReproStack-AWS679f53fac002430cb0da5b7982bd2287Se-OU09VCPOTDJ
W/EsReproStack-AWS679f53fac002430cb0da5b7982bd22872D-1FV5ZU8BVH07W is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-west-1:485225193788:key/65a4f72b-c
a0b-439e-bff0-1638de590be0 (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: 7a879401-561c-4feb-a1c4-4d72aa7cf09f; Proxy: null)
new CustomResource (/tmp/jsii-kernel-JNsVMr/node_modules/@aws-cdk/core/lib/custom-resource.js:27:25)
\_ new AwsCustomResource (/tmp/jsii-kernel-JNsVMr/node_modules/@aws-cdk/custom-resources/lib/aws-custom-resource/aws-custom-resource.js:164:31)
\_ new ElasticsearchAccessPolicy (/tmp/jsii-kernel-JNsVMr/node_modules/@aws-cdk/aws-elasticsearch/lib/elasticsearch-access-policy.js:14:9)
\_ new Domain (/tmp/jsii-kernel-JNsVMr/node_modules/@aws-cdk/aws-elasticsearch/lib/domain.js:737:34)
\_ /<path-to-python-env>/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3137:49
[████████████▉·············································] (2/9)
3:42:24 PM | CREATE_FAILED | Custom::ElasticsearchAccessPolicy | EsReproDomain/ESAc...y/Resource/Default
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::485225193788:assumed-role/EsReproStack-AWS679f53fac002430cb0da5b7982bd2287Se-OU09VCPOTDJ
W/EsReproStack-AWS679f53fac002430cb0da5b7982bd22872D-1FV5ZU8BVH07W is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-west-1:485225193788:key/65a4f72b-c
a0b-439e-bff0-1638de590be0 (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: 7a879401-561c-4feb-a1c4-4d72aa7cf09f; Proxy: null)
3:42:25 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | EsReproStack
The following resource(s) failed to create: [EsReproDomainESAccessPolicyDCE7667B]. Rollback requested by user.
❌ EsReproStack failed: Error: The stack named EsReproStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
at Object.waitForStackDeploy (/usr/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:305:11)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at Object.deployStack (/usr/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:283:26)
at CdkToolkit.deploy (/usr/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
at initCommandLine (/usr/lib/node_modules/aws-cdk/bin/cdk.ts:200:9)
The stack named EsReproStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
ignaloidas
on 23 Nov 2020
@iliapolo Just tested, still have the issue.
I think the issue occurs when combining the cluster with an access policy. Because the creation of the cluster doesn't fail, but the creation of the custom resource to apply the access policy fails.
The L2 construct creates a custom resources to set the access policy when the cluster is created: elasticsearch-access-policy.ts
Ruben-E
on 23 Nov 2020
@ignaloidas I see you also defined an access policy and it also fails on that. This error is exactly the same as I have
Ruben-E
on 23 Nov 2020
@Ruben-E @ignaloidas thanks for the info. I'll give that a try.
iliapolo
on 23 Nov 2020
@ignaloidas Are you able to consistently reproduce with the python example? Could you share which region are you deploying to?
iliapolo
on 23 Nov 2020
@iliapolo Yes, it is consistent, I'm deploying to eu-west-1.
ignaloidas
on 23 Nov 2020
For me its also consistent. Tested a couple of times today. Also eu-west-1
Ruben-E
on 23 Nov 2020
@Ruben-E do you have a minimal repro as well? the more the merrier
iliapolo
on 23 Nov 2020
@Ruben-E i mean something more detailed than what you previously posted (couldn't repro with that)
iliapolo
on 23 Nov 2020
@ignaloidas Are you sure the repro you posted is accurate? I'm hitting this error:
You must choose an even number of data nodes for a two Availability Zone deployment
because you specify 1 data node and using 2 availability zones (the default). Not that it matters for our repro, but i just want to make sure its all accurate.
Thanks
iliapolo
on 23 Nov 2020
I understand. I'll look at that tomorrow.
This was a piece from a bigger project which I'm not allowed to share, hence the small snippet 😉
Ruben-E
on 23 Nov 2020
because you specify 1 data node and using 2 availability zones (the default). Not that it matters for our repro, but i just want to make sure its all accurate.
Huh, no, I do specify to not have any availability zone awareness with zone_awareness=ZoneAwarenessConfig(enabled=False),
ignaloidas
on 23 Nov 2020
@ignaloidas oh right, apologies, copy paste error 👍
iliapolo
on 23 Nov 2020
It seems that in some, still unknown, conditions, the es:updateElasticsearchDomainConfig invoked by the ElasticsearchAccessPolicy requires the kms:DescribeKey permission on the key.
I'm reluctant to add this by default while its still unclear, but to get around this issue you can add the necessary permissions like so:
import * as cr from '@aws-cdk/custom-resources';
const key = new kms.Key(...);
const domain = new elastic.Domain(...);
const policyHandler = domain.node.tryFindChild('ESAccessPolicy') as cr.AwsCustomResource;
policyHandler.grantPrincipal.addToPrincipalPolicy(new iam.PolicyStatement({
actions: ['kms:DescribeKey'],
resources: [key.keyArn],
effect: iam.Effect.ALLOW,
}))
@Ruben-E this still doesn't explain why the same problem is happening in your case where only the L1 is being used.
iliapolo
on 23 Nov 2020
@iliapolo Thanks for the workaround, I'm going to give it a try.
Although I use the L1, I still use the ElasticSearchAccessPolicy custom resource (because ARN is needed for policies which is only available after deploy). Since the problem is not in the L1 / L2, but in the custom resource, I'm also affected.
This code also reproduces the issue and gives an idea how I set it up:
class Elasticsearch extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const key = new Key(this, 'Key', {enabled: true});
const domain = new CfnDomain(this, 'Domain', {
domainName: 'test',
elasticsearchVersion: ElasticsearchVersion.V7_7.version,
elasticsearchClusterConfig: {
instanceCount: 1,
instanceType: 't3.small.elasticsearch'
},
ebsOptions: {ebsEnabled: true, volumeSize: 80},
nodeToNodeEncryptionOptions: {enabled: true},
encryptionAtRestOptions: {enabled: true, kmsKeyId: key.keyId},
domainEndpointOptions: {enforceHttps: true, tlsSecurityPolicy: TLSSecurityPolicy.TLS_1_2},
})
const accessPolicy = new ElasticsearchAccessPolicy(this, 'AccessPolicy', {
domainName: domain.domainName!!,
domainArn: domain.attrArn,
accessPolicies: [allowHttp(domain.attrArn)]
});
accessPolicy.node.addDependency(domain);
function allowHttp(clusterArn: string) {
return new PolicyStatement({
effect: Effect.ALLOW,
actions: ['es:ESHttp*'],
principals: [new AnyPrincipal()],
resources: [Lazy.stringValue({produce: () => `${clusterArn}/*`})],
});
}
}
}
const app = new App();
new Elasticsearch(app, 'repro', stackProperties)
Output:
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:
IAM Statement Changes
┌───┬──────────────────────────────────────────────────────────┬────────┬──────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┬───────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼──────────────────────────────────────────────────────────┼────────┼──────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┼───────────┤
│ + │ ${CustomResourceLambdaRole.Arn} │ Allow │ sts:AssumeRole │ Service:lambda.amazonaws.com │ │
├───┼──────────────────────────────────────────────────────────┼────────┼──────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┼───────────┤
│ + │ ${Domain.Arn} │ Allow │ es:UpdateElasticsearchDomainConfig │ AWS:${CustomResourceLambdaRole} │ │
├───┼──────────────────────────────────────────────────────────┼────────┼──────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┼───────────┤
│ + │ ${Key.Arn} │ Allow │ kms:CancelKeyDeletion │ AWS:arn:${AWS::Partition}:iam::<accountid>:root │ │
│ │ │ │ kms:Create* │ │ │
│ │ │ │ kms:Delete* │ │ │
│ │ │ │ kms:Describe* │ │ │
│ │ │ │ kms:Disable* │ │ │
│ │ │ │ kms:Enable* │ │ │
│ │ │ │ kms:GenerateDataKey │ │ │
│ │ │ │ kms:Get* │ │ │
│ │ │ │ kms:List* │ │ │
│ │ │ │ kms:Put* │ │ │
│ │ │ │ kms:Revoke* │ │ │
│ │ │ │ kms:ScheduleKeyDeletion │ │ │
│ │ │ │ kms:TagResource │ │ │
│ │ │ │ kms:UntagResource │ │ │
│ │ │ │ kms:Update* │ │ │
└───┴──────────────────────────────────────────────────────────┴────────┴──────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┴───────────┘
IAM Policy Changes
┌───┬─────────────────────────────┬────────────────────────────────────────────────────────────────────────────────┐
│ │ Resource │ Managed Policy ARN │
├───┼─────────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${CustomResourceLambdaRole} │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole │
└───┴─────────────────────────────┴────────────────────────────────────────────────────────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)
Do you wish to deploy these changes (y/n)? y
repro: deploying...
[0%] start: Publishing 4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956:current
[100%] success: Published 4a3609ad912843e581892f37ae9d6fb0fa1648b547693aaa562b0119452b8956:current
repro: creating CloudFormation changeset...
[█████████████████████▊····································] (3/8)
7:48:34 AM | CREATE_FAILED | Custom::ElasticsearchAccessPolicy | AccessPolicy/Resource/Default
Failed to create resource. Error in Accessing KmsKeyID with details:User: arn:aws:sts::<accountid>:assumed-role/repro-CustomResourceLambdaRoleC810B87F-1IXJ7WNCEE7ZS/repro-AWS679f53fac002430cb0da5b7982bd22
872D164C4C-2IHWOHZLOAT8 is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:eu-west-1:<accountid>:key/901432c0-463b-4d9c-9120-f7daeaac01d6 (Service: AWSKMS; Status Code: 400; Error Code
: AccessDeniedException; Request ID: 3f07ab1d-e87c-4894-87ba-503a78f8f8d4; Proxy: null)
7:48:34 AM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | repro
The following resource(s) failed to create: [AccessPolicyD18466CD]. Rollback requested by user.
7:48:34 AM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | repro
The following resource(s) failed to create: [AccessPolicyD18466CD]. Rollback requested by user.
7:48:43 AM | DELETE_IN_PROGRESS | AWS::Elasticsearch::Domain | Domain
Can confirm, the workaround works
Ruben-E
on 24 Nov 2020
Can confirm, the workaround works
Awesome, i'm going to mark this as a bug and keep investigating.
Thanks
iliapolo
on 24 Nov 2020
@Ruben-E @ignaloidas @laimonassutkus managed to reproduce, thanks for all the cooperation :)
iliapolo
on 25 Nov 2020
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
![github-actions[bot] picture](https://avatars2.githubusercontent.com/in/15368?v=4&s=40)
github-actions[bot]
on 29 Dec 2020
Related issues
AlexCheema
·
32Comments
amirsch
·
58Comments
eladb
·
33Comments
markusl
·
37Comments
farzad-xgen
·
30Comments
Most helpful comment
It seems that in some, still unknown, conditions, the
es:updateElasticsearchDomainConfiginvoked by theElasticsearchAccessPolicyrequires thekms:DescribeKeypermission on the key.I'm reluctant to add this by default while its still unclear, but to get around this issue you can add the necessary permissions like so:
@Ruben-E this still doesn't explain why the same problem is happening in your case where only the L1 is being used.