Amplify-cli: Amplify ElasticSearch Streaming function breaks after AWS upgrades your Lambda runtime
Describe the bug
If you are getting the following error in your Amplify ElasticSearch streaming function (DdbToEsFn)
Unable to import module 'python_streaming_function': cannot import name 'BotocoreHTTPSession'
It is because AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. This causes problems with BotocoreHTTPSession due to changes made to it and can no longer be imported.
To reproduce your AWS account must be using the latest Python run time AWS is upgrading to or replicate the above environment on your local machine.
To Reproduce
Steps to reproduce the behavior:
- Assuming your account has the latest runtime, use any version 3.8 or lower of the cli
- Make sure one of your schema has
@searchable - Execute amplify --push
- Execute any mutation against the schema with
@searchablein it - Inspect logs of the DdbToEsFn to see the error
Expected behavior
Not to cause errors
tluolamo
All 30 comments
It suddenly resolved itself, so I guess this was issue on my AWS account
tluolamo
on 19 Sep 2019
Actually this kept on reoccurring and after talking to AWS support this is what they came back with
AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. All supported SDK features and APIs in this update are compatible with previous versions. As long as your function uses features and APIs documented in the SDK developer guide [1], no action is required. If you are using non-documented features or classes of the SDK you may potentially experience compatibility issues. One notable example of an internal class that underwent change is the BotocoreHTTPSession class which is no longer supported starting with botocore version 1.11.0. If your function is incompatible with this change, you can follow the best practice of including your version of boto3 and botocore as part of your function's deployment package instead of relying on the default SDK in Lambda's execution environment. Alternatively, you can update your function to use a Lambda layer [2] that contains the previous version of the SDK.
The steps for creating a deployment package in python are explained in [3].
Therefore to mitigate this issue, I would like to suggest a workaround for now. The AWS Lambda team had created the following layers which can be used with our functions in case the function gets impacted as a result of the above update (when importing the packages such as above - BotocoreHTTPSession).
For the functions in us-east-1, the following layers can be added based upon the runtime configured for the function:
python 2.7 and 3.6: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1
python 3.7: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:2
It seems viable as a work around so far by manually adding the layer, but I think Amplify should either include that layer as part of the push or update the function to work with the above version of boto.
tluolamo
on 20 Sep 2019
This is also happening to us since yesterday at some point. First started in our development environment, and somewhat solved itself but then started doing the same in production. This affects all our data as our frontfacing UI is using Elasticsearch to find the data.
How can this just happen? How can there be no internal coordination in AWS and maintainers of things like this?
houmark
on 20 Sep 2019
@houmark Yeah this should definitely be extremely high priority fix as pretty much everybody using Amplify will be impacted by this as AWS upgrades their Lambda run times. Our prod environment also broke last night as well, which is on a separate AWS account, luckily for us we are just short of being in production really with our system, so no real users were impacted this time.
tluolamo
on 20 Sep 2019
This was a planned change it seems, but it was so badly communicated, that nobody inside the AWS Amplify team was aware and had a chance to push out a fix. And even if they did, then this will require a push by every client using Elasticsearch on Amplify as far as I understand. Knowing AWS they will just release this in 3.8.x or 4.x and not care about 3.0.0, which means we are forced to upgrade to that version and potentially face other issues with multi auth etc.
The AWS and especially the AWS Amplify team needs to rethink their strategies on releases and keep 1-2 versions back which they do critical bugfixes on, even after a new version has recently been released. We used to jump on a new version right away, and especially before we were in production, but after having learned the hard way more than once, I am now holding back until I can see that the new versions seems mostly smooth. Also having no release notes since 3.0.0 doesn't make me feel like even attempting the update.
I've applied the layer and it seems to work, but little by little we have more and more and more patches on top of the main setup.
houmark
on 20 Sep 2019
@tluolamo This issue should be fixed from the Lambda service side as they rolled-back their changes - causing this behavior. Please feel free to comment on this thread if you're still seeing this issue.
kaustavghosh06
on 23 Sep 2019
Amazon just keeps the ball rolling on doing unstable stuff breaking the WORLD! Oh well...
houmark
on 23 Sep 2019
I still have the "workaround" ARN layer on top of my services, I hope the rollback does not mean that that now breaks my setup? Can you confirm if having the layer workaround is "safe"?
houmark
on 23 Sep 2019
@kaustavghosh06 this was for you, sorry, forgot to mention you...
houmark
on 23 Sep 2019
@kaustavghosh06 we haven't implemented the workaround. We are still seeing the issue in both our staging and production instances. Is there a certain amount of time to wait until Amazon is able to roll this back everywhere?
keithmacklin
on 23 Sep 2019
@keithmacklin Could you specify your Lambda ARN + region and send it across to [email protected]?
kaustavghosh06
on 23 Sep 2019
@houmark Let me update you regarding the layer workaround removal in a bit.
kaustavghosh06
on 23 Sep 2019
@kaustavghosh06 We added the "workaround" in our environments after I posted my comment and things are now working fine. I would also like to know if the workaround would continue to be safe. In the meantime, I will email you my ARN
keithmacklin
on 23 Sep 2019
@keithmacklin @houmark @tluolamo We dug into the issue and after talking to the Lambda team, we understood the reason for the change and errors. We're working on a fix in the CLI that will enable you to re-deploy your function as a part of the ampliy push command to avoid seeing the Lambda errors. For the time being, here's an explanation and the workaround provided by the Lambda team to prevent your app from seeing errors:
_AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. One notable example of an internal class that underwent change is the BotocoreHTTPSessions class which is no longer supported starting with botocore version 1.11.0._
Since the CLI used the BotocoreHTTPSessions class, that's the reason why you're seeing these errors.
As a workaround, you can update your DdbToEsFn-xxxxxxfunction in the AWS Console to use a Lambda layer that contains the previous version of the SDK. For your convenience, the AWS Lambda team already published these layers for you to use. You can find additional information on Lambda layers out here - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
Here are the list of lambda layer ARN's which you can use based on the region you are in:_
ap-northeast-1: arn:aws:lambda:ap-northeast-1:249908578461:layer:AWSLambda-Python-AWS-SDK:1
us-east-1: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-1: arn:aws:lambda:ap-southeast-1:468957933125:layer:AWSLambda-Python-AWS-SDK:1
eu-west-1: arn:aws:lambda:eu-west-1:399891621064:layer:AWSLambda-Python-AWS-SDK:1
us-west-1: arn:aws:lambda:us-west-1:325793726646:layer:AWSLambda-Python-AWS-SDK:1
ap-east-1: arn:aws:lambda:ap-east-1:118857876118:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-2: arn:aws:lambda:ap-northeast-2:296580773974:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-3: arn:aws:lambda:ap-northeast-3:961244031340:layer:AWSLambda-Python-AWS-SDK:1
ap-south-1:631267018583: arn:aws:lambda:ap-south-1:631267018583:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-2: arn:aws:lambda:ap-southeast-2:817496625479:layer:AWSLambda-Python-AWS-SDK:1
ca-central-1: arn:aws:lambda:ca-central-1:778625758767:layer:AWSLambda-Python-AWS-SDK:1
eu-central-1: arn:aws:lambda:eu-central-1:292169987271:layer:AWSLambda-Python-AWS-SDK:1
eu-north-1: arn:aws:lambda:eu-north-1:642425348156:layer:AWSLambda-Python-AWS-SDK:1
eu-west-2: arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1
eu-west-3: arn:aws:lambda:eu-west-3:959311844005:layer:AWSLambda-Python-AWS-SDK:1
sa-east-1: arn:aws:lambda:sa-east-1:640010853179:layer:AWSLambda-Python-AWS-SDK:1us-us-east-2: arn:aws:lambda:us-east-2:259788987135:layer:AWSLambda-Python-AWS-SDK:1
us-west-2: arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python-AWS-SDK:1
cn-north-1: arn:aws-cn:lambda:cn-north-1:683298794825:layer:AWSLambda-Python-AWS-SDK:1
cn-northwest-1: arn:aws-cn:lambda:cn-northwest-1:382066503313:layer:AWSLambda-Python-AWS-SDK:1
us-gov-west-1:: arn:aws-us-gov:lambda:us-gov-west-1:556739011827:layer:AWSLambda-Python-AWS-SDK:1
us-gov-east-1: arn:aws-us-gov:lambda:us-gov-east-1:138526772879:layer:AWSLambda-Python-AWS-SDK:1
@kaustavghosh06 in case anyone else missed it like it did, eu-west-1 is there:
'u-west-1: arn:aws:lambda:eu-west-1:399891621064:layer:AWSLambda-Python-AWS-SDK:1'
missing an e at the start
oliverandersencox
on 24 Sep 2019
@oliverandersencox Thanks for catching that. I've made the correction in my response above.
kaustavghosh06
on 24 Sep 2019
Hi Guys,
From 20th Sept, 2019 I started getting below error for my lambda function which read the data from dynamodb stream and post it to AWS elasticsearch. Below is the error. Is this cause by AWS lambda upgrade? Are they going to rollback or provide any fix? Any help/suggestion would be highly appreciated.
('ERROR: ', 'Traceback (most recent call last):\n File "/var/task/lambda_function.py", line 276, in lambda_handler\n return _lambda_handler(event, context)\n File "/var/task/lambda_function.py", line 270, in _lambda_handler\n post_to_es(es_payload) # Post to ES with exponential backoff\n File "/var/task/lambda_function.py", line 71, in post_to_es\n session = botocore.session.Session({'region': es_region})\n File "/var/runtime/botocore/session.py", line 127, in init\n self.session_var_map.update(session_vars)\n File "/usr/lib64/python2.7/_abcoll.py", line 566, in update\n self[key] = other[key]\n File "/var/runtime/botocore/session.py", line 938, in setitem\n self._update_config_store_from_session_vars(key, value)\n File "/var/runtime/botocore/session.py", line 959, in _update_config_store_from_session_vars\n config_name, env_vars, default, typecast = config_options\nValueError: too many values to unpack\n')
singhmaheshk
on 25 Sep 2019
Hey everyone,
I had this problem and I followed the solution that @kaustavghosh06 suggest and everything went well. Then I created a cloudformation template to fix this. I want to share this template with you guys, in case it might help someone. You can add this template to your project under amplify/backend/api/{you_api_local_resource_name}/stacks.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "An auto-generated nested stack.",
"Metadata": {},
"Parameters": {
"AppSyncApiId": {
"Type": "String",
"Description": "The id of the AppSync API associated with this project."
},
"AppSyncApiName": {
"Type": "String",
"Description": "The name of the AppSync API",
"Default": "AppSyncSimpleTransform"
},
"env": {
"Type": "String",
"Description": "The environment name. e.g. Dev, Test, or Production",
"Default": "NONE"
},
"DbToESFnName": {
"Type": "String",
"Default": "DdbToEsFn"
},
"PythonAWSSDKLayer": {
"Type": "String",
"Default": "arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1"
}
},
"Resources": {
"AppSyncFixDbToESFnExecutionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": {
"Fn::If": [
"HasEnvironmentParameter",
{
"Fn::Join": [
"",
[
{
"Ref": "AppSyncApiName"
},
"-",
"fix-db-to-es-fn-iam",
"-",
{
"Ref": "env"
}
]
]
},
{
"Fn::Join": [
"",
[
{
"Ref": "AppSyncApiName"
},
"-",
"fix-db-to-es-fn-iam"
]
]
}
]
},
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Policies": [
{
"PolicyName": "cloudWatchPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": {
"Fn::Sub": [
"arn:aws:logs:${region}:${account}:log-group:/aws/lambda/${lambda}:log-stream:*",
{
"region": {
"Ref": "AWS::Region"
},
"account": {
"Ref": "AWS::AccountId"
},
"lambda": {
"Fn::If": [
"HasEnvironmentParameter",
{
"Fn::Join": [
"",
[
{
"Ref": "AppSyncApiName"
},
"-",
"fix-db-to-es-fn",
"-",
{
"Ref": "env"
}
]
]
},
{
"Fn::Join": [
"",
[
{
"Ref": "AppSyncApiName"
},
"-",
"fix-db-to-es-fn"
]
]
}
]
}
}
]
}
}
]
}
},
{
"PolicyName": "lambdalayerPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lambda:UpdateFunctionConfiguration"
],
"Resource": {
"Fn::Sub": [
"arn:aws:lambda:${region}:${account}:function:${lambda}",
{
"region": {
"Ref": "AWS::Region"
},
"account": {
"Ref": "AWS::AccountId"
},
"lambda": {
"Fn::Join":[
"-",
[
{
"Ref":"DbToESFnName"
},
{
"Ref":"AppSyncApiId"
},
{
"Ref":"env"
}
]
]
}
}
]
}
},
{
"Effect": "Allow",
"Action": [
"lambda:GetLayerVersion"
],
"Resource": {
"Ref": "PythonAWSSDKLayer"
}
}
]
}
}
]
}
},
"AppSyncFixDbToESFnLambdaFunction": {
"DependsOn": [
"AppSyncFixDbToESFnExecutionRole"
],
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"ZipFile": {
"Fn::Join": [
"\n",
[
"const aws = require('aws-sdk');",
"exports.handler = async (event, context) => {",
" try {",
" console.log(event.RequestType);",
" console.log(event.ResourceProperties);",
" if (event.RequestType == 'Delete') {",
" await sendResponse(event, context, 'SUCCESS', {});",
" context.done();",
" }",
" else if (event.RequestType == 'Update' || event.RequestType == 'Create') {",
" const lambda = new aws.Lambda({ apiVersion: '2015-03-31', region: event.ResourceProperties.region });",
" const params = {",
" FunctionName: event.ResourceProperties.lambda,",
" Layers: [event.ResourceProperties.layer]",
" };",
" console.log(params);",
" await lambda.updateFunctionConfiguration(params).promise();",
" await sendResponse(event, context, 'SUCCESS', {});",
" context.done();",
" }",
" else {",
" console.log('No Event!!!');",
" context.done();",
" }",
" }",
" catch (err) {",
" console.log(err);",
" const responseData = { Error: err };",
" await sendResponse(event, context, 'FAILED', responseData);",
" context.done();",
" }",
"};",
"async function sendResponse(event, context, responseStatus, responseData, physicalResourceId, noEcho) {",
" var responseBody = JSON.stringify({",
" Status: responseStatus,",
" Reason: 'See the details in CloudWatch Log Stream: ' + context.logStreamName,",
" PhysicalResourceId: physicalResourceId || context.logStreamName,",
" StackId: event.StackId,",
" RequestId: event.RequestId,",
" LogicalResourceId: event.LogicalResourceId,",
" NoEcho: noEcho || false,",
" Data: responseData",
" });",
" console.log('Response body:', responseBody);",
" var https = require('https');",
" var url = require('url');",
" let parsedUrl;",
" try {",
" parsedUrl = url.parse(event.ResponseURL);",
" } catch (error) {",
" throw new Error(`Invalid url ${event.ResponseURL}`);",
" }",
" let options = {",
" hostname: parsedUrl.hostname,",
" port: 443,",
" path: parsedUrl.path,",
" method: 'PUT',",
" headers: {",
" 'content-type': '',",
" 'content-length': responseBody.length",
" }",
" };",
" return new Promise((resolve, reject) => {",
" const clientRequest = https.request(options, incomingMessage => { ",
" let response = {",
" statusCode: incomingMessage.statusCode,",
" statusMessage: incomingMessage.statusMessage,",
" headers: incomingMessage.headers,",
" body: []",
" };",
" incomingMessage.on('data', chunk => {",
" response.body.push(chunk);",
" });",
" incomingMessage.on('end', () => {",
" console.log('Status code: ' + response.statusCode);",
" console.log('Status message: ' + response.statusMessage);",
" try {response.body = response.body.join(); console.log('Response Body: ' + JSON.parse(response.body));} catch(e) {}",
" return resolve(response);",
" });",
" });",
" clientRequest.on('error', error => {",
" return reject(error);",
" });",
" if (responseBody) {",
" clientRequest.write(responseBody);",
" }",
" clientRequest.end();",
" });",
"}"
]
]
}
},
"Handler": "index.handler",
"FunctionName": {
"Fn::If": [
"HasEnvironmentParameter",
{
"Fn::Join": [
"",
[
{
"Ref": "AppSyncApiName"
},
"-",
"fix-db-to-es-fn",
"-",
{
"Ref": "env"
}
]
]
},
{
"Fn::Join": [
"",
[
{
"Ref": "AppSyncApiName"
},
"-",
"fix-db-to-es-fn"
]
]
}
]
},
"Role": {
"Fn::GetAtt": [
"AppSyncFixDbToESFnExecutionRole",
"Arn"
]
},
"Runtime": "nodejs8.10",
"Timeout": "300"
}
},
"AppSyncFixDbToESFnFunctionOutputs": {
"Type": "Custom::LambdaCallout",
"Properties": {
"ServiceToken": {
"Fn::GetAtt": [
"AppSyncFixDbToESFnLambdaFunction",
"Arn"
]
},
"region": {
"Ref": "AWS::Region"
},
"accountId": {
"Ref": "AWS::AccountId"
},
"lambda": {
"Fn::Join":[
"-",
[
{
"Ref":"DbToESFnName"
},
{
"Ref":"AppSyncApiId"
},
{
"Ref":"env"
}
]
]
},
"layer": {
"Ref": "PythonAWSSDKLayer"
}
}
}
},
"Conditions": {
"HasEnvironmentParameter": {
"Fn::Not": [
{
"Fn::Equals": [
{
"Ref": "env"
},
"NONE"
]
}
]
},
"AlwaysFalse": {
"Fn::Equals": [
"true",
"false"
]
}
},
"Outputs": {
"EmptyOutput": {
"Description": "An empty output. You may delete this if you have at least one resource above.",
"Value": ""
}
}
}
eelzinaty
on 25 Sep 2019
@kaustavghosh06 I still have the workaround in place and it seems to continue to work for me, I'm not planning to remove it anytime soon if this continues. Probably adding the workaround layer or creating a custom layer for the function would be the safest option for amplify to make sure the function doesn't suddenly break if this upgrade is pushed up again.
tluolamo
on 25 Sep 2019
In the latest version of the CLI - v3.10.0, we've added support for the above mentioned (https://github.com/aws-amplify/amplify-cli/issues/2353#issuecomment-534359395) Lambda Layer in to avoid breaking of the Lambda functions due to the Python Lambda runtime change. Please let us know if you're still seeing this issue.
kaustavghosh06
on 29 Sep 2019
Decided I would try to update amplify to get the benefit of the layer... now I cannot even push. Awesome. Just trading one bug for another.
Inaccessible host: xxxxxx-20190814102923-deployment.s3.amazonaws.com'. This service may not be available in theus-east-1' region.
mormsbee
on 22 Oct 2019
^I also had this issue. I updated to the lastest amplify version (npm i -g @aws-amplify/cli@latest) and it was fixed.
aehringer
on 23 Oct 2019
thank you for breaking my production app. This the third bug, that I had to fix. Truth is that you have caused me to lose my job amplify. I should have gone with firebase instead!
nagad814
on 5 Nov 2019
Seems this problem cropped up again for me - adding the layer seems to fix it.
However - what happens to all the missed DynamoDB events that occurred within that time period.
Is there a way to re-index/backfill - or should it happen automatically?
jonnyleeharris
on 13 Nov 2019
@jonnyleeharris it depends, if you fixed it in less than 24 hours all the changes would be queued and should be executed after the fix was put in place. But if it took you longer than that it is possible that some changes were lost. The DynamoDB streams that the feature is using keeps the messages available for at least 24 hours.
tluolamo
on 13 Nov 2019
@tluolamo definitely more than 24 hours. Guess I will have to look at manually synchronizing things? A lot of CRUD operations occurred outside of that 24 hour window. I had initially thought to scan all items of the table and modify them so the Lambda function would be triggered - not sure how to handle the deleted items though.
Thanks for your input.
jonnyleeharris
on 13 Nov 2019
@tluolamo Just checking, do you know how to increase the 24hours to longer time period?
cylim
on 21 Nov 2019
Good day.
I'm still getting the UserIllegalArgumentException "Elasticsearch responded with an error: Not Found" error even though I have the latest Amplify CLI 4.25.0 and I've checked the correct layer as listed above: eu-west-2: arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1
Any advise?
christodupreez
on 24 Jul 2020
Not found usually indicates there are is not matching index yet. Have you
added data?
On Fri, Jul 24, 2020, 7:49 AM Dupdroid notifications@github.com wrote:
Good day.
I'm still getting the UserIllegalArgumentException "Elasticsearch
responded with an error: Not Found" error even though I have the latest
Amplify CLI 4.25.0 and I've checked the correct layer as listed above:
eu-west-2:
arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1Any advise?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws-amplify/amplify-cli/issues/2353#issuecomment-663501104,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABUMX3D2P5Q34PXQW5DY2HTR5FYNFANCNFSM4IYP3NZQ
.
mormsbee
on 24 Jul 2020
Yes, I have added the index and also deleted the data and recreated.
Here is the error:
query NearbyBranches {
nearbyBranches(location: { lat: -34.030082, lon: 22.813462 },
m: 500,
limit: 1
) {
items { id location { lat lon } }
total
nextToken
}
}
{
"data": {
"nearbyBranches": null
},
"errors": [
{
"path": [
"nearbyBranches"
],
"data": null,
"errorType": "Elasticsearch:UserIllegalArgumentException",
"errorInfo": null,
"locations": [
{
"line": 2,
"column": 3,
"sourceName": null
}
],
"message": "Elasticsearch responded with an error: Not Found"
}
]
}
Related issues
nicksmithr
·
3Comments
jkeys-ecg-nmsu
·
3Comments
jexh
·
3Comments
kangks
·
3Comments
ffxsam
·
3Comments
Most helpful comment
@keithmacklin @houmark @tluolamo We dug into the issue and after talking to the Lambda team, we understood the reason for the change and errors. We're working on a fix in the CLI that will enable you to re-deploy your function as a part of the ampliy push command to avoid seeing the Lambda errors. For the time being, here's an explanation and the workaround provided by the Lambda team to prevent your app from seeing errors:
_AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. One notable example of an internal class that underwent change is the BotocoreHTTPSessions class which is no longer supported starting with botocore version 1.11.0._
Since the CLI used the
BotocoreHTTPSessions class, that's the reason why you're seeing these errors.As a workaround, you can update your
DdbToEsFn-xxxxxxfunction in the AWS Console to use a Lambda layer that contains the previous version of the SDK. For your convenience, the AWS Lambda team already published these layers for you to use. You can find additional information on Lambda layers out here - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.htmlHere are the list of lambda layer ARN's which you can use based on the region you are in:_