Aws-cdk: Service Linked Role Requred for ElasticSearch Domain Not Created Automatically

[jrjohnson]

:bug: Bug Report

What is the problem?

CfnDomain does not create a service linked role automatically. I think this makes is impossible to deploy reliably using only CDK as the role may have been created by a previous domain and can't be created twice in the same region. The only option seems to be pre-creating the role in the CLI before deploying which is awkward to document and difficult (maybe impossible) to automate.

Reproduction Steps

const vpc = new Vpc(this, "Vpc", {
    maxAzs: 3
});
new CfnDomain(scope, 'Search', {
    elasticsearchVersion: '6.7',
    elasticsearchClusterConfig: {
      instanceCount: 2,
      zoneAwarenessEnabled: true,
      instanceType: 't2.small.elasticsearch',
    },
    ebsOptions: {
      ebsEnabled: true,
      volumeSize: 10,
      volumeType: 'gp2'
    },
    vpcOptions: {
      securityGroupIds: [
        vpc.vpcDefaultSecurityGroup
      ],
      subnetIds: vpc.privateSubnets.map(subnet => subnet.subnetId),
    }
  });

Verbose Log

$ npm run cdk deploy
...
15/27 | 2:04:49 PM | CREATE_FAILED        | AWS::Elasticsearch::Domain            | Search Before you can proceed, you must enable a service-linked role to give Amazon ES permissions to access your VPC. (Service: AWSElasticsearch; Status Code: 400; Error Code: ValidationException;)
...

Environment

• CDK CLI Version: 1.4.0 (build 175471f)
• Module Version: @aws-cdk/aws-elasticsearch - 1.4.0
• OS: OSX
• Language: TypeScript

Related to #569 which indicates that these roles should be built automatically.

[nija-at]

Hello @jrjohnson -

Thanks for filing the report.

Unfortunately, CDK doesn't have full support for ElasticSearch support. You can find our support and stability notice here. We only have the autogenerated low-level constructs in them. We've not gotten around to implementing the higher level constructs. (See here for the different types of constructs).

The low-level constructs match one to one with CloudFormation resources, which is why CfnDomain doesn't create an IAM service role, and will only attempt to create an ElasticSearch Domain.

We are looking for contributions from the community to fill in the gaps and we'll happy to look at PRs that add in higher level constructs to our ElasticSearch module.

Thanks!

[ottokruse]

Workaround to DIY in CDK:

const serviceLinkedRole = new cdk.CfnResource(this, "es-service-linked-role", {
  type: "AWS::IAM::ServiceLinkedRole",
  properties: {
    AWSServiceName: "es.amazonaws.com",
    Description: "Role for ES to access resources in my VPC"
  }
});

const esDomain = new es.CfnDomain(this, "es", { ... });

esDomain.node.addDependency(serviceLinkedRole);

[logemann]

thanks @ottokruse. I really struggled to attach the ServiceLinkedRole to esDomain. Looked for a CDK class representing the linkedRole. grrr.

The overall topic should be a featReq then or? I really would like to see higher level constructs for ES. Perhaps i will find the time to do something there but still need to wrap my head around the whole cdk project internally.

Update.: get the same error even with that addDependency()

[hopenbr]

@logemann @ottokruse
Did you ever get the workaround to be successful

I too am in the same boat, I still get the below error even about adding ServiceLinkedRole

_"Before you can proceed, you must enable a service-linked role to give Amazon ES
permissions to access your VPC"_

code:

CfnResource serviceLinkedRole = CfnResource.Builder.create(this,"es-service-linked-role")    

.type("AWS::IAM::ServiceLinkedRole")

.properties( Map.ofEntries(entry("AWSServiceName", "es.amazonaws.com"),                                                                                      
                                           entry("Description", "Role for ES to access")))
.build();

... 

es.addDependsOn(serviceLinkedRole);

[ottokruse]

The solution I posted worked for me. But apparently not for @logemann

[hopenbr]

I get the ES to spin up if I run

aws iam create-service-linked-role --aws-service-name es.amazonaws.com 

I have to remove cdk code from stack, to get it to work.

[logemann]

Yeah same for me. Very unfortunate.

[ottokruse]

TBH the service linked role has to be created once (and only once) in an AWS account, not for every ES deployment. So it does make sense to either create it using a one time CLI command, or using CloudFormation in a separate base stack.

My guess is that after creating the Service Linked Role there might be a bit of time needed somtimes, for ES to "see" that it is there and use it. That might explain why my solution worked for me (and some others I guess since I see thumbs up) but not for all. Instantiating the service linked role in a base stack would practically also solve the issue then.

[jrjohnson]

I think it is under appreciated how difficult a one off job like creating this role is in a fully automated provisioning setup. Especially if there is an arbitrary delay before it seems to work. Our setup requires us to be able to reliably build and deploy to a new region (or entirely new account) quickly. Having a single manual step in an otherwise automated process is a huge foot gun. It's safer for us to just execute a well written set of instructions instead, validating on each step.

This single issue is completely blocking our migration to CDK. Given that this was closed without resolution a year ago I thought there might be a long term plan to add support for all services in CDK. I guess not? Telling us to build it ourselves is a weird stance to take from a company that makes billions of dollars and for a service we pay for. I'm happy to take a crack at it if AWS is willing to foot the bill for development time or provide my university free service in exchange.

[logemann]

You are right with your point that closing this case is wrong because it doesnt work for a lot of users as it seems. Furthermore doing stuff like this manually contradicts the whole concept of CDK. I can live with manually creating a Client VPN Endpoint but Roles should really work in order to get deploy step running.

[spicysomtam]

I am suffering this issue. Not sure what has changed in aws. Tried to different accounts:

$ aws iam create-service-linked-role --aws-service-name es.amazonaws.com 
An error occurred (InternalFailure) when calling the CreateServiceLinkedRole operation (reached max retries: 4): Unknown

Even trying to create an es domain manually on the console reports an error:

Amazon Elasticsearch Service can't create the service-linked role. Try again, and if you continue to have problems, contact AWS Support.

Google throws up nothing.Guess I will need to log a support ticket with aws?

[spicysomtam]

GO to log a support ticket, and even before I log one, this is flagged!

Increased API Error Rates
We have identified the root cause of the increased error rates and latencies on the AWS IAM CreateRole and CreateServiceLinkedRole APIs and are working towards resolution. Other AWS services such as AWS CloudFormation whose features require these actions may also be impacted. User authentications and authorizations are not impacted.

[ambasta]

@nija-at ES L2 client was merged in #8369

However, the L2 Client also lacks the ability to autocreate service linked role. Is this expected behavior?