Amplify-cli: amplify does not create indices on ES

[waltermvp]

* Which Category is your question related to? *
API

* What AWS Services are you utilizing? *
Appsync + DynamoDB + ElasticSearch Service

* Provide additional details e.g. code snippets *
When creating test and production environments amplify push fails to create indices+mappings on the new ES domains. My dev instance does have the correct indices + mappings.

I have tried to remove @searchable then add it again (per model) with no luck. Not sure where to begin looking to see if this is a user or amplify error. It is likely user error since this did work when setting up my dev environments and much work has been done on the project since. Thank you

[elorzafe]

Hi @waltermvp this is issue is related to Amplify CLI, I will transfer it to that repo.

Thanks!

[RossWilliams]

Elastic determines property mappings the first time it sees data. It is likely when you dev environment was setup the first objects it saw had certain data types that are different from when your production environment first received data.

If you want to setup elastic with specific mappings, you would need to either 1. Manually setup mappings when your ES cluster is created, or 2. Use a custom resource to setup mappings automatically when an ES cluster is created.

Amplify can help with option 2 by placing custom resources in the amplify/backend/api/PROJ_NAME/stacks folder as cloudformation templates. When you push these resources will be created alongside the amplify generated resources. Documentaion describes this functionality here.

Here is a stripped back cloudformation template I have used to set mappings (please verify this suits your needs and review roles and permissions are suitable for your situation).

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
        "AppSyncApiId": {
            "Type": "String",
            "Description": "The id of the AppSync API associated with this project."
        },
        "AppSyncApiName": {
            "Type": "String",
            "Description": "The name of the AppSync API",
            "Default": "AppSyncSimpleTransform"
        },
        "env": {
            "Type": "String",
            "Description": "The environment name. e.g. Dev, Test, or Production",
            "Default": "NONE"
        }
    },
    "Resources": {
        "ConfigESCustom": {
            "Type": "Custom::ConfigureES",
            "Properties": {
                "ServiceToken": {
                    "Fn::GetAtt": [
                        "ConfigureES",
                        "Arn"
                    ]
                }
            }
        },
        "ConfigureES": {
            "Type": "AWS::Lambda::Function",
            "Properties": {
                "Environment": {
                    "Variables": {
                        "ES_ENDPOINT": {
                            "Fn::ImportValue": {
                                "Fn::Join": [
                                    ":",
                                    [
                                        {
                                            "Ref": "AppSyncApiId"
                                        },
                                        "GetAtt",
                                        "Elasticsearch",
                                        "DomainEndpoint"
                                    ]
                                ]
                            }
                        },
                        "ES_REGION": {
                            "Ref": "AWS::Region"
                        },
                        "DEBUG": 1,
                        "ELASTICSEARCH_DOMAIN_ARN": {
                            "Fn::ImportValue": {
                                "Fn::Join": [
                                    ":",
                                    [
                                        {
                                            "Ref": "AppSyncApiId"
                                        },
                                        "GetAtt",
                                        "Elasticsearch",
                                        "DomainArn"
                                    ]
                                ]
                            }
                        }
                    }
                },
                "Code": {
                    "ZipFile": "import base64\nimport json\nimport logging\nimport string\nimport boto3\nimport os\nimport time\nimport datetime\nimport traceback\nfrom urllib.parse import urlparse, quote\nfrom botocore.vendored import requests\nfrom botocore.auth import SigV4Auth\nfrom botocore.awsrequest import AWSRequest\nfrom botocore.credentials import get_credentials\nfrom botocore.endpoint import BotocoreHTTPSession\nfrom botocore.session import Session\nimport cfnresponse\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n\n# The following parameters are required to configure the ES cluster\nES_ENDPOINT = os.environ['ES_ENDPOINT']\nES_REGION = os.environ['ES_REGION']\nDEBUG = True if os.environ['DEBUG'] is not None else False\n\ndef es_put(payload, region, creds, host, path, method='PUT', proto='https://'):\n  '''Put index data to ES endpoint with SigV4 signed http headers'''\n  req = AWSRequest(method=method, url=proto + host +\n    quote(path), data=payload, headers={'Host': host, 'Content-Type': 'application/json'})\n  SigV4Auth(creds, 'es', region).add_auth(req)\n  http_session = BotocoreHTTPSession()\n  res = http_session.send(req.prepare())\n  return res._content\n\ndef lambda_handler(event, context):\n  logger.info('got event {}'.format(event))\n\n  if event['RequestType'] == 'Create':\n    # Get aws_region and credentials to post signed URL to ES\n    es_region = ES_REGION or os.environ['AWS_REGION']\n    session = Session({'region': es_region})\n    creds = get_credentials(session)\n    es_url = urlparse(ES_ENDPOINT)\n    # Extract the domain name in ES_ENDPOINT\n    es_endpoint = es_url.netloc or es_url.path\n    es_put('', es_region, creds,\n      es_endpoint, '/INDEX_NAME')\n    es_actions = []\n    es_actions.append('')  # Add one empty line to force final \\n\n    es_payload = '\\n'.join(es_actions)\n    action = {\"properties\": {\"MY_PROPERTY\": {\"type\": \"MY_TYPE\"}}}\n    es_actions.append(json.dumps(action))\n    es_actions.append('')  # Add one empty line to force final \\n\n    es_payload = '\\n'.join(es_actions)\n    es_put(es_payload, es_region, creds,\n      es_endpoint, '/INDEX_NAME/_mapping/doc')\n\n  cfnresponse.send(event, context, cfnresponse.SUCCESS, {})\n"
                },
                "FunctionName": {
                    "Fn::Join": [
                        "-",
                        [
                            "ConfigureES",
                            {
                                "Ref": "env"
                            }
                        ]
                    ]
                },
                "Handler": "index.lambda_handler",
                "Timeout": 30,
                "Role": {
                    "Fn::GetAtt": [
                        "LambdaRole",
                        "Arn"
                    ]
                },
                "Runtime": "python3.6"
            }
        },
        "LambdaRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    "lambda.amazonaws.com"
                                ]
                            },
                            "Action": [
                                "sts:AssumeRole"
                            ]
                        }
                    ]
                },
                "Path": "/",
                "Policies": [
                    {
                        "PolicyName": "lambda-logs",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "logs:CreateLogGroup",
                                        "logs:CreateLogStream",
                                        "logs:PutLogEvents"
                                    ],
                                    "Resource": [
                                        "arn:aws:logs:*:*:*"
                                    ]
                                }
                            ]
                        }
                    },
                    {
                        "PolicyName": "ES",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "es:ESHttpPut"
                                    ],
                                    "Resource": {
                                        "Fn::Join": [
                                            "",
                                            [
                                                {
                                                    "Fn::ImportValue": {
                                                        "Fn::Join": [
                                                            ":",
                                                            [
                                                                {
                                                                    "Ref": "AppSyncApiId"
                                                                },
                                                                "GetAtt",
                                                                "Elasticsearch",
                                                                "DomainArn"
                                                            ]
                                                        ]
                                                    }
                                                },
                                                "/*"
                                            ]
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        },
        "Outputs": {
            "EmptyOutput": {
                "Description": "An empty output. You may delete this if you have at least one resource above.",
                "Value": ""
            }
        }
    }
}

[kaustavghosh06]

@RossWilliams Thanks for the response. Agreed with what you're recommending reagrding custom resources to set the mappings. @waltermvp Let us know if you need further help.

[waltermvp]

@kaustavghosh06 This is the direction i was looking for, feel free to close this issue thank you