Amplify-cli: [Feature] Export the Elasticseach Domain Endpoint and Arn in the SearchableStack

[kstro21]

Is your feature request related to a problem? Please describe.
Currently, the SearchableStack has 0 outputs defined. I'm trying to use the Elasticsearch DomainEndpoint in a custom Lambda added using amplify function add but I had to wait for the service to be created, go to the AWS Elasticsearch Service to copy the value and hardcode it in the Lambda template. Not ideal when we can import the value.

Describe the solution you'd like
Add outputs to the SearchableStack and export the DomainArn and DomainEndpoint.

Additional context
I'm using amplify 1.1.7

[mikeparisstuff]

Hey @kstro21. Yes, I don't see any issues with adding the Elasticsearch details to the exports and should be a relatively quick task. The relevant code is to add Outputs here: https://github.com/aws-amplify/amplify-cli/blob/0665508833fbedc0e1a787a90d8b72435fd71696/packages/graphql-elasticsearch-transformer/src/resources.ts#L82-L93. I would be happy to review a PR if you would like to contribute and otherwise will add this to the backlog.

[kstro21]

@mikeparisstuff that would be great. I'll be back with a PR. Thanks

[kstro21]

Hey @mikeparisstuff. Well, it turns out not to be a quick task, at least for me. I've added the changes in the file you suggested and 2 files more. It looks good, but, the outputs are not written to the template amplify\backend\api\amplifytest\build\stacks\SearchableStack.json and instead to amplify\backend\api\amplifytest\build\cloudformation-template.json, additionally, the output is generated using "Fn::ImportValue" thus failing when running amplify push. Here is an example of what is generated.

"ElasticsearchDomainArn": {
    "Description": "Elasticsearch instance Domain ARN.",
    "Value": {
        "Fn::ImportValue": {
            "Fn::Join": [
                ":",
                [
                    {
                        "Ref": "AppSyncApiId"
                    },
                    "GetAtt",
                    "ElasticSearchDomain",
                    "DomainArn"
                ]
            ]
        }
    },
    "Export": {
        "Name": {
            "Fn::Join": [
                ":",
                [
                    {
                        "Ref": "AppSyncApiId"
                    },
                    "GetAtt",
                    "Elasticsearch",
                    "DomainArn"
                ]
            ]
        }
    }
},

The changes are simple, I would like you to take a look at my changes here https://github.com/kstro21/amplify-cli/commit/ce71a5aaaae33570a0876fa9b6bc32a330a10cb9 .

I did some debug and I find out some interesting things. I didn't see code to map the outputs to stacks like exists for the resources https://github.com/aws-amplify/amplify-cli/blob/767f3a6edce9f8c4109e18afacf648c6d4c7be56/packages/graphql-transformer-core/src/util/splitStack.ts#L53-L72

When the function collectTemplates is invoked, it is only passing to the template the Parameters, Conditions and Resources thus leaving the Outputs attached to the root stack.
https://github.com/aws-amplify/amplify-cli/blob/767f3a6edce9f8c4109e18afacf648c6d4c7be56/packages/graphql-transformer-core/src/util/splitStack.ts#L80-L92
https://github.com/aws-amplify/amplify-cli/blob/767f3a6edce9f8c4109e18afacf648c6d4c7be56/packages/graphql-transformer-core/src/util/splitStack.ts#L105-L107

The code generating the "Fn::ImportValue" inside the outputs config is in the funcion https://github.com/aws-amplify/amplify-cli/blob/767f3a6edce9f8c4109e18afacf648c6d4c7be56/packages/graphql-transformer-core/src/util/splitStack.ts#L116

As I don't want to add any extra/unneeded code, I have to ask first: Am I missing some settings for this to work? Can you please, point me the right direction? Or, should I implement the logic to map the Outputs to stacks like the following function? https://github.com/aws-amplify/amplify-cli/blob/767f3a6edce9f8c4109e18afacf648c6d4c7be56/packages/graphql-transformer-core/src/util/splitStack.ts#L53

I'll be looking forward to your response

[mikeparisstuff]

@kstro21 Thanks for taking a look at this. I need to look into this a little deeper but I think you are correct and we will need to map the outputs to the stacks. If you have a fork with most of the changes implemented, I could try pulling from there and take a look.

[kstro21]

@mikeparisstuff I have a branch searchable-stack-outputs in my fork here https://github.com/kstro21/amplify-cli/tree/searchable-stack-outputs you can see my changes in this commit https://github.com/kstro21/amplify-cli/commit/ce71a5aaaae33570a0876fa9b6bc32a330a10cb9 the changes are only to add the outputs to the SearchableStack. I can add a POC of a function to map the outputs to the stacks if you like then you can start taking a look from there.

Just let me know

[kstro21]

@mikeparisstuff I have a working version of the code in my fork, mapping outputs with the stacks. I've tested on an AWS account with success.

If you want to look into this a little deeper by reviewing my changes, feel free to hit me back with the feedback, I would like to make a PR. The code is in https://github.com/kstro21/amplify-cli/tree/searchable-stack-outputs

I just would like that we could have a more friendly export name like described in #986 but that is another story.

[sollipse]

Hey guys, after updating to the newest CLI, updates to our Amplify stack are failing with the message:

AWS::CloudFormation::Stack Sat Apr 06 2019 15:55:53 GMT-0700 (PDT) Template format error: Output ElasticsearchDomainArn is malformed. The Name field of Export must not depend on any resources, imported values, or Fn::GetAZs.

[kstro21]

Hi, @sollipse, sorry to hear that you are having issues. I'm using the latest version(1.5.1) without any issue.

Could this be a cache issue? Probably npm keeping a stale version of a package causing the issue? Can you uninstall @aws-amplify/cli and do a fresh install of it?

Let me know if that helps

[mwarger]

@kstro21 I'm getting this same issue as @sollipse after re-installing the CLI.

[kstro21]

@mwarger @sollipse could you share the step to reproduce this, please? I've just grabbed a brand new virtual machine, installed amplify 1.5.1 and created a new env without any issues

As you can see in the SS, the stack has been created and the outputs are correct.

What is your OS? What amplify version is being installed globally or locally?

The only way the searchable stack is added is by adding the @searchable directive to one of your model in your schema.graphql.

[mwarger]

I already have an elasticsearch domain setup - this only happened after upgrading the CLI to the latest version and attempting to push a pre-existing project.

[kstro21]

@mwarger I have done that with success on pre-existing projects.

What is your OS? What amplify version is being installed? Globally or locally?
Could you check what version of the package graphql-transformer-core is being installed inside @aws-amplify/cli and share it?

[mwarger]

@kstro21

OS: Windows 10
Amplify version installed globally: 1.5.1
graphql-transformer-core: 3.4.0

[mwarger]

@kstro21 @mikeparisstuff Any thoughts if this is caused by a change to cloudformation or another backend? I've tried downgrading my version of the amplify/cli without luck. Pretty stumped by this one...

Thanks for any assistance you can provide.

[kstro21]

@kstro21

OS: Windows 10
Amplify version installed globally: 1.5.1
graphql-transformer-core: 3.4.0

@mwarger I'm the same env. Just yesterday we updated an env in production using version 1.5.1 without problems.

Have you copied any template from build\stacks to stacks?

Can you share only the outputs of [myproject]\amplify\backend\api\[myapi]\build\cloudformation-template.json and [myproject]\amplify\backend\api\[myapi]\build\stacks\SearchableStack.json after running amplify api gql-compile?

[sollipse]

Hey @kstro21 , I deleted my CLI, reinstalled, and still encounter this error.

@mwarger , until this is fixed I've been just deleting the segments with the ElasticSearchDomainArn from the generated CF template during the compile step of amplify push. This results in a successful push, though I've had to repeat that step every time I want to push new code.

Currently working in an application where it would be prohibitively expensive to drop the stack and repush.

[sollipse]

@kstro21 , while I appreciate that it works in your dev environment, the existence of errors encountered by at least 2 other amplify customers suggests this issue should be reopened for now

[mwarger]

@sollipse - I had to delete the following two sections from the cloudformation-template.json to get it to push successfully.

"ElasticsearchDomainArn": {
"Description": "Elasticsearch instance Domain ARN.",
"Value": {
"Fn::GetAtt": [
"ElasticSearchDomain",
"DomainArn"
]
},
"Export": {
"Name": {
"Fn::Join": [
":",
[
{
"Ref": "AppSyncApiId"
},
"GetAtt",
"Elasticsearch",
"DomainArn"
]
]
}
}
},
"ElasticsearchDomainEndpoint": {
"Description": "Elasticsearch instance Domain Endpoint.",
"Value": {
"Fn::Join": [
"",
[
"https://",
{
"Fn::GetAtt": [
"ElasticSearchDomain",
"DomainEndpoint"
]
}
]
]
},
"Export": {
"Name": {
"Fn::Join": [
":",
[
{
"Ref": "AppSyncApiId"
},
"GetAtt",
"Elasticsearch",
"DomainEndpoint"
]
]
}
}
}

[mwarger]

@kstro21 Here are my files that you requested.

contents.zip

[kstro21]

Hey, @mwarger, that's weird. Those two sections shouldn't be in cloudformation-template.json. Are we talking about the cloudformation-template.json that is autogenerated in the build\ folder? Actually, those two sections should be in the build\stacks\SearchableStack.json.

Weirder is that I've reviewed the 2 files you attached(thx for that) and the definition for the "AWS::Elasticsearch::Domain" is in the cloudformation-template.json and not in the SearchableStack.json. How did you get to that? Are you using some custom logic to generate the templates?

Hey, @mikeparisstuff, is this an old format generated from an old version of the cli? I'm talking about the definition for the "AWS::Elasticsearch::Domain" located in the cloudformation-template.json and not in the SearchableStack.json. Is this something that can be migrated without breaking the env?

@sollipse @mwarger what version of the cli were you using before upgrading to version 1.5.1?
Also, can you share what version of NodeJS are you using?
What is your version of Typescript?
And once again, are you using some custom logic to modify the template before/after generate it? If you do, can you share it?

I'm trying to reproduce this but I'm having a bad time with no success.

[mwarger]

I'm not doing anything custom.

I'm not sure what version of the CLI I was using before I upgraded, but it's been a couple weeks since I did an update.

node -v
v10.15.3
tsc -v
Version 3.4.2

Again, nothing custom. Files attached are what I had - if there's something odd about them, it's what I got after running the compile (and the same results I get when doing amplify push).

Thanks for looking into this!

[kstro21]

Hy, @mwarger, I can not reproduce the issue with the information provided. It would be great to know the version used before upgrading to the latest.

Can you do a litter test? No need to amplify push nothing.

1. Create a new folder
2. cd into the new folder
3. Run amplify init
4. Run amplify api add and add a new GrahQL API
5. Edit your schema.graphql and add the @searchable directive to one model
6. Run amplify api gql-compile
7. Share the [newfolder]\amplify\backend\api\[myapi]\build\cloudformation-template.json and [newfolder]\amplify\backend\api\[myapi]\build\stacks\SearchableStack.json files.

Hey, @mikeparisstuff any thoughts why the definition for the "AWS::Elasticsearch::Domain" is located in the cloudformation-template.json and not in the SearchableStack.json?

All I can do right now is start testing all the versions before 1.6.1 was released and hope to reproduce the issue.

[kstro21]

@sollipse - I had to delete the following two sections from the cloudformation-template.json to get it to push successfully.

@mwarger by the way, how you did to edit the cloudformation-template.json? AFAIK, it is autogenerated and cannot be edited, right? What is the path of that file that you edited?

[mwarger]

@kstro21 it can be edited after the compilation is done, but before you press Y to accept whether you want to do code generation for your queries. You can save the file, continue, and then it will use the saved version.

[mwarger]

@kstro21 I don't believe I have any way to tell what version I had before upgrading.

[mikeparisstuff]

Hey, @mikeparisstuff any thoughts why the definition for the "AWS::Elasticsearch::Domain" is located in the cloudformation-template.json and not in the SearchableStack.json?

@kstro21 The ES domain would be in root stack if the project was created with an earlier version of the CLI. When we moved to nested stacks, the existing resources that contain data remained in the top level so they were not deleted and data would not be lost. To do so a file is created transform.conf.json that store some info telling us which resources should remain in the root stack.

The transform package is informed of these overrides here in the GraphQLTransform constructor here https://github.com/aws-amplify/amplify-cli/blob/master/packages/graphql-transformer-core/src/GraphQLTransform.ts#L186. I believe a fix is to adjust the stack mappings for the outputs that were added in the PR such that if the ES domain is being overridden to remain in the root, so too will the output.

[kstro21]

@mikeparisstuff that is a good start. Please, can you tell me an old version of the cli that I can use to reproduce the issue? The hunt is driving me crazy, testing one version after another one. If I'm able to reproduce the issue, I know I can fix it, and I want to.

[mwarger]

@kstro21 @mikeparisstuff This is indeed a project that I have shepherded through multiple versions of the CLI over time.

[mwarger]

@kstro21 @mikeparisstuff I wanted to follow up and see if there has been any progress on this? I can get by for now in my local environment by altering the cloudformation-template, but this is still preventing me from pushing to my remote environment, as I cannot make the same alterations to the template in the amplify console.

Thank you!

[kstro21]

Hi, @mwarger, sorry for the delay, I've been a little short of time but already did some tests using version 0.2.1 and I was able to reproduce the issue. I'll be back with the fix soon.

Regards

[kstro21]

Happy to open #1316 with the fix.

[nagey]

I'm still seeing this same issue with 1.6.5… I've tried to remove and reinstall the CLI with the same situation (similarly downgrading back down to 1.1.8 which didn't have this problem). @sollipse's fix (manually deleting the bad bits) works for me here as well.

[kstro21]

Hey, @nagey, sorry to hear that you are having issues. The code with the fix was merged into master, but it is not released yet.

[nagey]

@kstro21 ah — that makes sense. Figured since it had been merged a few days back, and 1.6.5 came out yesterday, that this fix was in it. Do you know what version this fix is targeting?

[kstro21]

@nagey, sorry, but that is a question for @mikeparisstuff, @kaustavghosh06 or maybe @yuth

[yuth]

@nagey @kstro21 we are planning to do an release today and this change should be in that release

[mwarger]

@yuth Was this released? I'm still getting this error in the amplify console. Thanks!

[kaustavghosh06]

@mwarger Probably amplify console hasn’t updated the CLI used in the build process to the latest version. To get around this immediately, you could install the latest CLI as a part of your build script. That way the build process will use the latest version of the CLI. Nevertheless, I’ll inform the console team to update the default CLI version.

[mwarger]

@kaustavghosh06 @yuth How do I install the CLI during the build step? I've added it to the preBuild step of the backend, and it appears to install the 1.6.6 build - however I'm still getting this error. Please advise.

[mwarger]

I tried this again, and added the -g flat to the install. My latest build worked.