Boto3: How to import boto3==1.1.4 on AWS Lambda

[vlcinsky]

I am trying to use boto3==1.1.4 on AWS Lambda.

I did not find a way to import this version, even though I have in included in the zip archive.

Running the code locally it reports version 1.1.4

On AWS Lambda it reports only version 1.1.3.

Trying to request specific version by means of pkg_resources.require("boto3==1.1.4") does not improve the situation.

This is my code:

from lxml import etree
from py.path import local
import json
import gzip
import pkg_resources
pkg_resources.require('boto3==1.1.4')
import boto3

print(boto3.__version__)
assert boto3.__version__ == "1.1.4"

s3 = boto3.resource("s3")


def fetch(bucket_name, key_name, version_id, fname):
    keyver = s3.ObjectVersion(bucket_name, key_name, version_id).Object()

    assert "download_file" in dir(keyver)
    keyver.download_file(fname.strpath)
    return fname.strpath


def transform(fname):
    doc = etree.parse(fname.strpath)
    ids = doc.xpath("//ElaboratedRecord/@id")
    data = json.dumps(ids)
    return data


def publish(data, bucket_name, key_name, tmpdir):
    fname = tmpdir / "es.json.gz"

    try:
        with gzip.open(fname.strpath, "wb") as gf:
            gf.write(data)
        extra_args = {"ContentType": "application/json",
                      "ContentEncoding": "gzip"}
        s3.Object(bucket_name, key_name).upload_file(fname.strpath,
                                                     ExtraArgs=extra_args)
    finally:
        fname.remove()


def main(event, context):
    assert len(event["Records"]) == 1
    s3rec = event["Records"][0]["s3"]
    bucket_name = s3rec["bucket"]["name"]
    key_name = s3rec["object"]["key"]
    version_id = s3rec["object"]["versionId"]

    tmpdir = local.mkdtemp()
    infname = tmpdir / "es.xml.gz"
    try:
        fetch(bucket_name, key_name, version_id, infname)
        data = transform(infname)
        pubbucket = "sandbox.dp.ce-traffic.com"
        pubkey = "region/cz/EventService.json"
        publish(data, pubbucket, pubkey, tmpdir)
    finally:
        tmpdir.remove(rec=1, ignore_errors=True)

When I run this code, log reports:

START RequestId: 1fbc70e5-76a6-11e5-bb30-f17c9f6ee1a6 Version: $LATEST
module initialization error: (boto3 1.1.3 (/var/runtime), Requirement.parse('boto3==1.1.4'))

END RequestId: 1fbc70e5-76a6-11e5-bb30-f17c9f6ee1a6

This is probably more related to AWS Lambda environment itself, but I have no other place to report this problem.

[mtdowling]

We can also take a look, but you can reach out to the Lambda team directly on their forums at https://forums.aws.amazon.com/forum.jspa?forumID=186.

[vlcinsky]

Thanks @mtdowling

I have found description of my problem in the forum: https://forums.aws.amazon.com/thread.jspa?threadID=217554&tstart=0

• it is considered a bug, which shall be resolved and resolution reported to given thread
• there is simple workaround mentioned there.

As this is not related to boto3 but to AWS Lambda, I consider this issue to be closed (in boto3 tracker).

Btw, the workaround looks as follows:

import os
import os.path
import sys

root = os.environ["LAMBDA_TASK_ROOT"]
sys.path.insert(0, root)
import boto3 #should grab boto3 from included zip first 

[byumark]

This still seems to be an issue. AWS says it's possible and they don't mention a workaround:
http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Note AWS Lambda includes the AWS SDK for Python (Boto 3), so you don't need to include it in your deployment package. However, if you want to use a version of Boto3 other than the one included by default, you can include it in your deployment package.

Should I address this issue with AWS?

[vlcinsky]

@byumark Good idea to remind AWS that they shall either resolve the bug or include description of workaround in the documentation.
I did not check with the latest version of lambda, but if the bug still persist and you have an option to talk to AWS, address it. It shall improve the situation.

[byumark]

@vlcinsky Looks like I was wrong, sorry. I had my modules, including boto3, inside of a modules directory in my project. That was the issue.

[secureoptions]

If you don't want to package a more recent boto3 version with you function, you can download boto3 with each invocation of the Lambda. Remember that /tmp/ is the directory that Lambda will allow you to download to, so you can use this to temporarily download boto3:

import sys
from pip._internal import main

main(['install', 'boto3', '--target', '/tmp/'])
sys.path.insert(0,'/tmp/')

import boto3
from botocore.exceptions import ClientError

def handler(event, context):
    print(boto3.__version__)