Aws-sam-cli: boto3 not available in local Python3.6

[kaukas]

Description

boto3 seems to not be available when running lambda functions locally through sam local invoke.

Steps to reproduce

$ sam init # Choose template 1, runtime 9 - python3.6, quick start template 1
$ cd sam-app
$ sam local invoke
Invoking app.lambda_handler (python3.6)
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.0.0 image. Invoking with the already downloaded image.
Mounting /private/tmp/sam/sam-app/hello_world as /var/task:ro,delegated inside runtime container
28 Jul 2020 08:40:36,629 [INFO] (/var/runtime/awslambda/bootstrap.py) main started at epoch 1595925636629
28 Jul 2020 08:40:36,639 [INFO] (/var/runtime/awslambda/bootstrap.py) init complete at epoch 1595925636639
START RequestId: fb3c2a3d-7b37-135c-0f0c-7aaee074068a Version: $LATEST
END RequestId: fb3c2a3d-7b37-135c-0f0c-7aaee074068a
REPORT RequestId: fb3c2a3d-7b37-135c-0f0c-7aaee074068a  Init Duration: 157.34 ms    Duration: 2.69 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 26 MB

{"statusCode":200,"body":"{\"message\": \"hello world\"}"}
$ # All good so far
$ echo 'import boto3' >> hello_world/app.py
$ sam local invoke
Invoking app.lambda_handler (python3.6)
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.0.0 image. Invoking with the already downloaded image.
Mounting /private/tmp/sam/sam-app/hello_world as /var/task:ro,delegated inside runtime container
28 Jul 2020 08:42:07,219 [INFO] (/var/runtime/awslambda/bootstrap.py) main started at epoch 1595925727219
Unable to import module 'app': No module named 'boto3'

START RequestId: fe1c19d9-d036-1c9d-88e6-ee8ebc90ffd7 Version: $LATEST
END RequestId: fe1c19d9-d036-1c9d-88e6-ee8ebc90ffd7
REPORT RequestId: fe1c19d9-d036-1c9d-88e6-ee8ebc90ffd7  Init Duration: 168.50 ms    Duration: 2.02 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 26 MB
28 Jul 2020 08:42:07,235 [INFO] (/var/runtime/awslambda/bootstrap.py) init complete at epoch 1595925727235

{"errorMessage":"InvalidErrorShape"}

This used to work with earlier images.

Observed result

Unable to import module 'app': No module named 'boto3'

Expected result

boto3 is supposed to be available in Lambda environments.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac OS
2. sam --version: SAM CLI, version 1.0.0

Add --debug flag to command you are running

$ sam local invoke --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
Found one Lambda function with name 'HelloWorldFunction'
Invoking app.lambda_handler (python3.6)
No environment variables found for function 'HelloWorldFunction'
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=/private/tmp/sam/sam-app, CodeUri=hello_world/
Resolved absolute path to code is /private/tmp/sam/sam-app/hello_world
Code /private/tmp/sam/sam-app/hello_world is not a zip/jar file
Failed to download image with name amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.0.0
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.0.0 image. Invoking with the already downloaded image.
Mounting /private/tmp/sam/sam-app/hello_world as /var/task:ro,delegated inside runtime container
Starting a timer for 3 seconds for function 'HelloWorldFunction'
28 Jul 2020 08:43:49,216 [INFO] (/var/runtime/awslambda/bootstrap.py) main started at epoch 1595925829217
Unable to import module 'app': No module named 'boto3'

START RequestId: 336d2134-ad65-1030-bef9-c97c7a0a8844 Version: $LATEST
END RequestId: 336d2134-ad65-1030-bef9-c97c7a0a8844
REPORT RequestId: 336d2134-ad65-1030-bef9-c97c7a0a8844  Init Duration: 161.05 ms    Duration: 1.99 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 26 MB
28 Jul 2020 08:43:49,229 [INFO] (/var/runtime/awslambda/bootstrap.py) init complete at epoch 1595925829230
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 5927, 'exitReason': 'success', 'exitCode': 0, 'requestId': '055268b8-b43a-4822-b371-31abd291f9d9', 'installationId': '6a3ba03d-489c-46e4-8636-3c88565736ff', 'sessionId': '42e27585-9444-4a23-8fe6-8ac3774be51e', 'executionEnvironment': 'CLI', 'pyversion': '3.7.8', 'samcliVersion': '1.0.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

{"errorMessage":"InvalidErrorShape"}

[pbaezab]

+1

[danish45007]

+1

[vardhmanhundia]

+1

[BigMountainTiger]

+1

[johnc44]

Been meaning to post this, but been a bit swamped. It works fine on sam 0.53.0, so that's where I'm staying for the time being.

[S4lem]

+1

[aaronbrighton]

+1, not clear how to downgrade to 0.53.0 using Homebrew, reinstalled using pip instead.

[BigMountainTiger]

I noticed that the message - "Failed to download a new ...." is actually a false alarm.

The "local invoke" actually works without a problem. I have tried this in a totally new virtual machine, and made sure that I did not have any cached images. All of my "local invoke" and "local start-api" worked.

May be in the next release, they can remove this false alarm. For now, I think that we can just ignore it.

[mbuotidem]

@BigMountainTiger Do your lambdas use boto3? If you're not importing boto3, they might work fine without your running into this issue.

While the OP is on Mac, I'm on Windows and installed version 1.0.0 on WSL1 the homebrew way set out on the docs page. I also separately installed version 1.0.0 on the Windows side. Both were giving me the same Unable to import module 'app': No module named 'boto3' error. I got rid of all my currently existing docker images and tried again, same error.

On my WSL1, I uninstalled the homebrew version and installed 0.53.0 using pip3 install aws-sam-cli==0.53.0 and it worked. Got the 0.53.0 version for Windows from : https://github.com/awslabs/aws-sam-cli/releases/tag/v0.53.0 and that works as well.

[BigMountainTiger]

@mbuotidem

Actually a quick try succeeded:

_import boto3

def lambda_handler(event, context):

s3 = boto3.client('s3')
buckets = s3.list_buckets()
print(buckets['Buckets'])

return {
'Payload': {
'Time': 'Success'
}
}_

It works nicely, although I still see the message - "Failed to download a new amazon/aws-sam-cli-emulation-image-python3.8 ....."

with ... SAM CLI, version 1.0.0

[awood45]

This should be fixed for all Python invoke images. Please reopen this issue if you see this issue recur. You do not need to upgrade your version of AWS SAM CLI to see the changes on this. However, if you do see the issue recur, please remove your Docker images associated with Python runtime emulation, in case it failed to auto-update.

[nardonef]

@awood45 Still getting this on, sam start api, tried re pulling the docker images

[johnc44]

@awood45 - upgraded to 1.1.0 and it's worse for me. I now get this when I call a locally running API:

'Invoking main.lambda_handler (python3.6)\n'
b'Image was not found.\n'

So going back to 0.53.0. Not sure if this is just a new issue or whether it's just aprt of the same thing.

[mrpatrick]

installed from scratch and getting this same error

[awood45]

Reopening while we try to reproduce issues with the new images.

[awood45]

Question for those still seeing issues: Can you please return the output of the following command?

docker images | grep emulation

For me, I get the following:

amazon/aws-sam-cli-emulation-image-python3.7                               rapid-1.1.0         64f45dd252ec        22 hours ago        889MB
amazon/aws-sam-cli-emulation-image-python2.7                               rapid-1.1.0         5d468741d15c        23 hours ago        719MB
amazon/aws-sam-cli-emulation-image-python3.6                               rapid-1.1.0         354b29a6f806        23 hours ago        849MB
amazon/aws-sam-cli-emulation-image-python2.7                               latest              02d844ce4296        3 days ago          706MB
amazon/aws-sam-cli-emulation-image-python3.6                               latest              bced403d14ab        3 days ago          836MB
amazon/aws-sam-cli-emulation-image-python3.7                               latest              705267e12e30        3 days ago          876MB

One of our operating theories is that older images are getting cached somehow. The output of this command will help us to confirm/deny that hypothesis.

[awood45]

@johnc44 Can you give more detailed reproduction instructions? That looks like a separate issue but I'd like to try and reproduce that ASAP. I'm also interested in your docker images | grep emulation output.

[mbuotidem]

@awood45 I switched back to 0.53.0 but just to help out I installed 1.0.0 again. This is the result of docker images | grep emulation before installing 1.0.0.

amazon/aws-sam-cli-emulation-image-python3.6   latest              38f590981e83        7 weeks ago         776MB

And this is the result after installing it:

amazon/aws-sam-cli-emulation-image-python3.6   rapid-1.0.0         ca376949b68f        23 seconds ago      849MB
amazon/aws-sam-cli-emulation-image-python3.6   latest              bced403d14ab        3 days ago          836MB
amazon/aws-sam-cli-emulation-image-python3.6   <none>              38f590981e83        7 weeks ago         776MB

So yes, It no longer throws the Unable to import module 'app': No module named 'boto3' error, but I think there are probably still issues with the latest image. For example, I tried to do s3.buckets.all() and kept getting botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid. It took me a while to realize that there probably was still an issue with the image which a test from my non docker python confirmed. I promptly rolled back to 0.53.0 and the error was gone there as well.

[johnc44]

@awood45 - just tried upgrading to 1.1.0 again and it is now working. But for reference here's the info you requested:

Before upgrading to 1.1.0 again I ran docker images | grep emulation

amazon/aws-sam-cli-emulation-image-python3.6 rapid-1.1.0 aeb97e10d7c5 2 days ago 849MB
amazon/aws-sam-cli-emulation-image-python3.6 latest bced403d14ab 5 days ago 836MB
amazon/aws-sam-cli-emulation-image-python3.6 rapid-1.0.0 fb26df5453c7 3 weeks ago 790MB
amazon/aws-sam-cli-emulation-image-python3.6 38f590981e83 8 weeks ago 776MB

Now I've reinstalled 1.1.0 and done the same, I get the exact same output.

To get the error I was using "sam build && sam local start-api -n environment.json -p 3100"

Not sure if I was doing something stupid last week, because it looks like it brought down the image. I'll monitor it this afternoon but hopefully whatever it was is fixed. Thanks for your speedy reply and apologies if it was something my end.

[awood45]

@mbuotidem What kind of credentials are you using when you run local testing? That error does not look like it should be a Docker/image issue, and we've been trying without success to reproduce the error. Any other information you have about how you got that error would be helpful.

@johnc44 No worries, do let me know if it returns and we can try to reproduce.

[mbuotidem]

@awood45 I used AWS IAM credentials. Really I all I did was import boto3 inside app.py of the provided sam-app and then inside lambda_handler, I tried to list buckets with :

s3 = boto3.resource('s3')
buckets = list(s3.buckets.all())
print(buckets)

Since its not a Docker/image issue, I should probably just create a different issue so as not to derail this thread.

[michaelschem]

I'm still getting this issue after upgrading sam 1.0.0 -> 1.1.0

% sam --version
SAM CLI, version 1.1.0

% sam local invoke
Invoking app.lambda_handler (python3.7)
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.7:rapid-1.1.0 image. Invoking with the already downloaded image.
Mounting /Users/michaelschem/PycharmProjects/AWS Lambdas/.aws-sam/build/elasticsearchprocessalerts as /var/task:ro,delegated inside runtime container
START RequestId: 5c03f297-ff17-103b-4c06-605b6ec67fc8 Version: $LATEST

Here are my docker images for emulation.

% docker images | grep emulation
amazon/aws-sam-cli-emulation-image-python3.7       rapid-1.1.0                           7be905de0d0b        2 minutes ago       889MB
amazon/aws-sam-cli-emulation-image-python3.7       latest                                705267e12e30        2 weeks ago         876MB
amazon/aws-sam-cli-emulation-image-python3.7       rapid-1.0.0                           ac7eac4ecf37        3 weeks ago         887MB
amazon/aws-sam-cli-emulation-image-python3.8       rapid-1.0.0                           cd498fd504ba        4 weeks ago         503MB
amazon/aws-sam-cli-emulation-image-python3.8       latest                                0b8397b4f176        5 weeks ago         490MB
amazon/aws-sam-cli-emulation-image-python3.7       <none>                                da4c049dcbed        2 months ago        874MB

[awood45]

Can you delete your ‘:latest’ tag for the Python 3.7 emulation image and re-download? I believe your image might be cached.

[benjaminbours]

I have the same issue with SAM CLI, version 1.10.0.

Unable to import module 'app': No module named 'boto3'

Here is the output of docker images | grep emulation

amazon/aws-sam-cli-emulation-image-python3.6   latest                                cc52c542cdc2        8 weeks ago          844MB

Did anybody found a workaround? I can put boto3 in my requirements.txt file but it bundles 52M for botocore, I really would like to avoid this.

It's clearly written it should be available. https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html

EDIT: My bad I have realized it comes from my PYTHONPATH env variable.