Amplify-cli: Unable to stream models to ElasticSearch

[usmansbk]

What is the current behavior?
When I create a mutation, the lambda function that streams dynamodb models to elasticsearch fails and logs Unable to import module 'python_streaming_function': cannot import name 'BotocoreHTTPSession'.
I haven't edited any of the code as it is automatically generated by the GraphQL Transform

What is the expected behavior?
I should be able to search my models in AWS elasticsearch

Which versions and which environment (browser, react-native, nodejs) / OS are affected by this issue? Did this work in previous versions?

    "aws-amplify": "^1.1.40",
    "aws-amplify-react-native": "^2.1.19",
    "aws-appsync": "^2.0.0",
    "aws-appsync-react": "^2.0.0",
    "react": "^16.9.0",
    "react-apollo": "2.5.8",
    "react-native": "^0.60.5",

     aws-cli/1.16.243 Python/3.6.0 Windows/10 botocore/1.12.233

[sollipse]

+1 We have started seeing this issue in an extremely similar timeframe.

Has the version of Botocore made available to AWS Lambda changed? Alternatively, have the amplify team made any changes to the underlying zipfile housing this Streaming code?

Please treat this as a high severity issue. As Amplify does not offer a way to "refresh" entries in ES, the only remedy to missing ES records at them moment is a full table rewrite.

[kaustavghosh06]

We're looking into it. I'll update on this as soon as we have a fix for it. As a workaround please take a look at - #2353 (comment) for a workaround until we push the fix.

[kaustavghosh06]

This issue should be fixed from the Lambda service side as they rolled-back their changes - causing this behavior. Please feel free to comment on this thread if you're still seeing this issue.

[usmansbk]

@kaustavghosh06 still not fixed.

[kaustavghosh06]

@usmansbk We dug into the issue and after talking to the Lambda team, we understood the reason for the change and errors. We're working on a fix in the CLI that will enable you to re-deploy your function as a part of the ampliy push command to avoid seeing the Lambda errors. For the time being, here's an explanation and the workaround provided by the Lambda team to prevent your app from seeing errors:

_AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. One notable example of an internal class that underwent change is the BotocoreHTTPSessions class which is no longer supported starting with botocore version 1.11.0._

Since the CLI used the BotocoreHTTPSessions class, that's the reason why you're seeing these errors.

As a workaround, you can update your DdbToEsFn-xxxxxxfunction in the AWS Console to use a Lambda layer that contains the previous version of the SDK. For your convenience, the AWS Lambda team already published these layers for you to use. You can find additional information on Lambda layers out here - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

Here are the list of lambda layer ARN's which you can use based on the region you are in:_

ap-northeast-1: arn:aws:lambda:ap-northeast-1:249908578461:layer:AWSLambda-Python-AWS-SDK:1
us-east-1: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-1: arn:aws:lambda:ap-southeast-1:468957933125:layer:AWSLambda-Python-AWS-SDK:1
eu-west-1: arn:aws:lambda:eu-west-1:399891621064:layer:AWSLambda-Python-AWS-SDK:1
us-west-1: arn:aws:lambda:us-west-1:325793726646:layer:AWSLambda-Python-AWS-SDK:1
ap-east-1: arn:aws:lambda:ap-east-1:118857876118:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-2: arn:aws:lambda:ap-northeast-2:296580773974:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-3: arn:aws:lambda:ap-northeast-3:961244031340:layer:AWSLambda-Python-AWS-SDK:1
ap-south-1:631267018583: arn:aws:lambda:ap-south-1:631267018583:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-2: arn:aws:lambda:ap-southeast-2:817496625479:layer:AWSLambda-Python-AWS-SDK:1
ca-central-1: arn:aws:lambda:ca-central-1:778625758767:layer:AWSLambda-Python-AWS-SDK:1
eu-central-1: arn:aws:lambda:eu-central-1:292169987271:layer:AWSLambda-Python-AWS-SDK:1
eu-north-1: arn:aws:lambda:eu-north-1:642425348156:layer:AWSLambda-Python-AWS-SDK:1
eu-west-2: arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1
eu-west-3: arn:aws:lambda:eu-west-3:959311844005:layer:AWSLambda-Python-AWS-SDK:1
sa-east-1: arn:aws:lambda:sa-east-1:640010853179:layer:AWSLambda-Python-AWS-SDK:1us-us-east-2: arn:aws:lambda:us-east-2:259788987135:layer:AWSLambda-Python-AWS-SDK:1
us-west-2: arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python-AWS-SDK:1
cn-north-1: arn:aws-cn:lambda:cn-north-1:683298794825:layer:AWSLambda-Python-AWS-SDK:1
cn-northwest-1: arn:aws-cn:lambda:cn-northwest-1:382066503313:layer:AWSLambda-Python-AWS-SDK:1
us-gov-west-1:: arn:aws-us-gov:lambda:us-gov-west-1:556739011827:layer:AWSLambda-Python-AWS-SDK:1
us-gov-east-1: arn:aws-us-gov:lambda:us-gov-east-1:138526772879:layer:AWSLambda-Python-AWS-SDK:1

[sacrampton]

I am also seeing this problem - I created a new type with searchable and no index is created in ElasticSearch.

The instructions for the work around are a bit opaque to me - assuming you have just used standard amplify/appsync to sync DynamoDB and ElasticSearch, what are the specific set of steps/commands we need to execute to "unbreak" this whilst we are waiting for the fix?

[usmansbk]

@sacrampton

• Open the generated lambda function configuration in the AWS console -> Lambda, which is typically named DdbToEsFn-XXXXXX.

• Select layers
  

• Then click the Add a layer button and select the ARN option.

• Paste one of the ARNs above based on your region

[sacrampton]

Thanks for the above - I could add the layer - pushed to deploy again - but still no data streaming to ElasticSearch. I updated amplify CLI to latest version being pushing.

[tezqa]

@kaustavghosh06 Thanks for the workaround! I hope the AWS team will not do breaking changes like this one!

[ericclemmons]

For anyone finding this from Google because their GraphQL API queries (e.g. searchPhotos(filter: { description: { match: "foo } })) is returning Elasticsearch:UserIllegalArgumentException errors with the message:

Elasticsearch responded with an error: Not Found

This bug appears to be related. Subscribe to this thread & try out the fix listed here: https://github.com/aws-amplify/amplify-cli/issues/2385#issuecomment-534389117.

[cylim]

The issues should be related to https://github.com/boto/botocore/issues/1544

[babu-upcomer]

@kaustavghosh06 Why is it amplify CLI, JS, Cognito and Lambda teams are working out of sync? There has been hundreds of issues referring one another in the github itself. I do not want to highlight the feedback I get from the AWS support. To make the amplify a complete framework to serverless cloud problems you guys should work in sync.

Back to the issue, this issue has halted our ES indexing for the last 1 week and we got complaints from users that the search is no more working. Unfortunately this happened at our production env. In future, what level of guarantee we'll get to make sure that this doesn't happen again?

[bernhardt1]

@babu-upcomer thats rough.

I'm in beta right now and have a similar story (luckily didn't have more than a few dozen people's experience hurt). It kind of undermines my trust in AWS in general to have something like this happen.

[cylim]

I think lambda should warned developers who using build-in libraries from AWS, this is not only affecting Amplify + Elasticsearch users, but everyone who used deprecated library.

In the meantime, what Amplify can do is to specify version for each dependencies.

[kaustavghosh06]

In the latest version of the CLI - v3.10.0, we've added support for the above mentioned (https://github.com/aws-amplify/amplify-cli/issues/2353#issuecomment-534359395) Lambda Layer in to avoid breaking of the Lambda functions due to the Python Lambda runtime change. Please let us know if you're still seeing this issue.

[ArjunAhuja-git]

@kaustavghosh06 I am still seeing this issue in lamda function.

[avespoli-bf]

@kaustavghosh06 still seeing the issue even after trying the workaround

[nikhname]

@ArjunAhuja95 @avespoli-bf which version of the cli are you using?

[avespoli-bf]

@nikhname 3.11.0

[avespoli-bf]

@nikhname @kaustavghosh06 I didn't add the layer correctly to my lambda, it's working now.

[mikeRChambers610]

I am confused why this issue is closed. My Lambda function already has the layer which is being suggested to add as the fix. If I have the option to connect my App Sync API to an Elastic Search domain, it should work out of the box.

I am currently getting the below error when trying to connect to an already working Elastic Search domain.

Any help would be greatly appreciated! Thanks in advance!

{
"data": {
"listEvents": null
},
"errors": [
{
"path": [
"listEvents"
],
"data": null,
"errorType": "Elasticsearch:UserIllegalArgumentException",
"errorInfo": null,
"locations": [
{
"line": 2,
"column": 1,
"sourceName": null
}
],
"message": "Elasticsearch responded with an error: Forbidden"
}
]
}

[avespoli-bf]

@mikeRChambers610 Bear with me I haven't used appsync or amplify in over a year but I will try to help out. These are the arns you should be using. This problem should be resolved by now seeing how this issue is well over a year old but it appears that it isn't so this is how i fixed it. you should be using a lambda function to stream your models from your dynamo into elastic search. everything should be inside the same region. There was some sort of bug introduced into the built in streaming lambda SDKs which is why you need to swap out whatever amplify is using by default with an older version provided below.

What you need to do is

ap-northeast-1: arn:aws:lambda:ap-northeast-1:249908578461:layer:AWSLambda-Python-AWS-SDK:1
us-east-1: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-1: arn:aws:lambda:ap-southeast-1:468957933125:layer:AWSLambda-Python-AWS-SDK:1
eu-west-1: arn:aws:lambda:eu-west-1:399891621064:layer:AWSLambda-Python-AWS-SDK:1
us-west-1: arn:aws:lambda:us-west-1:325793726646:layer:AWSLambda-Python-AWS-SDK:1
ap-east-1: arn:aws:lambda:ap-east-1:118857876118:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-2: arn:aws:lambda:ap-northeast-2:296580773974:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-3: arn:aws:lambda:ap-northeast-3:961244031340:layer:AWSLambda-Python-AWS-SDK:1
ap-south-1:631267018583: arn:aws:lambda:ap-south-1:631267018583:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-2: arn:aws:lambda:ap-southeast-2:817496625479:layer:AWSLambda-Python-AWS-SDK:1
ca-central-1: arn:aws:lambda:ca-central-1:778625758767:layer:AWSLambda-Python-AWS-SDK:1
eu-central-1: arn:aws:lambda:eu-central-1:292169987271:layer:AWSLambda-Python-AWS-SDK:1
eu-north-1: arn:aws:lambda:eu-north-1:642425348156:layer:AWSLambda-Python-AWS-SDK:1
eu-west-2: arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1
eu-west-3: arn:aws:lambda:eu-west-3:959311844005:layer:AWSLambda-Python-AWS-SDK:1
sa-east-1: arn:aws:lambda:sa-east-1:640010853179:layer:AWSLambda-Python-AWS-SDK:1us-us-east-2: arn:aws:lambda:us-east-2:259788987135:layer:AWSLambda-Python-AWS-SDK:1
us-west-2: arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python-AWS-SDK:1
cn-north-1: arn:aws-cn:lambda:cn-north-1:683298794825:layer:AWSLambda-Python-AWS-SDK:1
cn-northwest-1: arn:aws-cn:lambda:cn-northwest-1:382066503313:layer:AWSLambda-Python-AWS-SDK:1
us-gov-west-1:: arn:aws-us-gov:lambda:us-gov-west-1:556739011827:layer:AWSLambda-Python-AWS-SDK:1
us-gov-east-1: arn:aws-us-gov:lambda:us-gov-east-1:138526772879:layer:AWSLambda-Python-AWS-SDK:1

1. Open the generated lambda function configuration in the AWS console -> Lambda, which is typically named DdbToEsFn- XXXXXX.

2. Select layers

3. Then click the Add a layer button and select the ARN option.

4. Paste one of the ARNs above based on your region (in your case it's `arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1`)

I would like to note that the error you are getting is due to the fact the index you're trying to access doesn't exist. This is most likely due to the fact that you can't (or you at least couldn't back when I was working with amplify last year) retroactively add already existing indices to your ES. If the index already exists inside the DB and the streaming functionality hasn't been set up yet, next time you go to add something to that index it will fail because as far as ES is concerned it doesn't exist. The easiest way to get around this is start from scratch with a fresh dynamo and ES instance. If that is not a possibility I'm sure you could write some sort of lambda that will go through your dynamo and add any missing indexes into your ES instance.

Good luck. It's stuff like this that made me ditch amplify completely. This kind of stuff should work flawlessly out of the box. There's way to much overhead and confusing configurations, not to mention pretty nasty bugs within the CLI back when I tried to use it. I'm hoping it matures someday but when I came across this issue over a year ago, this project was no where near ready to be used in a high volume production environment.