Web3.py: Web3.py on AWS Lambda

[greese]

• Version: current
• Python: 2.7
• OS: linux - was lambda

What was wrong?

Dependencies on cytoolz (which depends on a C compiler) make deployment on AWS lambda not work, whereas web3.js is able to deploy on AWS lambda.

The specific error is:

Unable to import module 'mylambda': /var/task/cytoolz/functoolz.so: undefined symbol: PyFPE_jbuf

How can it be fixed?

No clue. Not sure what the dependency on cytoolz does.

[carver]

I don't know much about the lambda options. Is it py2-only? If so, web3.py v4 won't work on it anyway, and v3 is in only being maintained with critical security patches.

This suggests that CPython was compiled without a necessary flag (not necessarily that cytoolz needs to be compiled):
https://stackoverflow.com/a/47703373/8412986

[greese]

Actually, I'm doing both Python 3.6 and Python 2.7 builds. I just reported the 2.7. Here's the 3.6 output from lambda:

Unable to import module 'sparq.lambdas.rates': No module named '_pysha3'

Now, the pysha3 stuff is actually installed:

(p3) ubuntu@host:~/deploy/p3$ find . -name \*pysha3\* -print
./lib/python3.5/site-packages/_pysha3.cpython-35m-x86_64-linux-gnu.so
./lib/python3.5/site-packages/pysha3-1.0.2.dist-info
(p3) ubuntu@host:~/deploy/p3$ pip install pysha3
Requirement already satisfied: pysha3 in ./lib/python3.5/site-packages

[carver]

Is sparq.lambdas.rates your module? It's a little weird that it's importing a sha3 module directly. If you need it for something, import sha3 instead of _pysha3 (but you shouldn't, you should just be able to use Web3.sha3()). Anyway, this may be some other thing unrelated to web3.py. What call is triggering the error? Do you still get the error if the only thing your code does is call Web3.sha3(text='testing')?

The full stack trace and code is needed to debug further.

I'm guessing some kind of module naming conflict issue, but don't have enough information.

[greese]

Yeah. I went ahead and created a new handler that looks like this:

from web3 import *

def lambda_handler(event, context):
    return Web3.sha3(text='testing')

And I got the same:

Unable to import module 'hello': No module named '_pysha3'

[greese]

As far as a stack trace, it's not even loading the handler module, so it's not getting to a stack trace point.

[carver]

Reading this SO post made me think about the environment. Maybe it can't import because it was built with the wrong target environment?

I'm trying to guess the target build environment based on the file name:

./lib/python3.5/site-packages/_pysha3.cpython-35m-x86_64-linux-gnu.so

I'm a newb at python C extensions, but I'm guessing the 35 in there means Python 3.5 and you mentioned that your Lambda is running in Py3.6. Everything else looks right, I assume Lambda it runs in linux, on a 64-bit cpu with cpython (not pypy).

[greese]

Good catch. I'll have a look at that.

[greese]

This may take longer than I'd like. For some reason, when installing ethereum, I am getting the no Python.h thing, but python3-dev is installed. Grrrr...

[carver]

Hm, well we're inching toward completely removing the dependency on pyethereum so keep an eye on #505 .

[greese]

OK, that last one was me failing to realize I needed a python3.6-dev specific to 3.6.

I am still getting the C compiler issue in Python 3:

Unable to import module 'sparq.lambdas.rates': /var/task/cytoolz/functoolz.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFPE_jbuf

[carver]

So back to my first comment: looks like you need a CPython that is compiled with --with-fpectl.

[greese]

This gets beyond my Python skill set. I've done some googling, but can't figure out how to do that. Any quick instructions? Thanks.

[carver]

I've never had to compile CPython on my machines, so I'm guessing it's actually a default setting. Either try to download the python binaries instead of using whatever is coming default, or you could try this guide (which I imagine works for 3.6 just fine as well):

https://passingcuriosity.com/2015/installing-python-from-source/

[greese]

Thanks!

Also, I found a more detailed description of the issue here: https://bugs.python.org/issue29137

I'll pass on exact instructions in this thread when I get this working. For now, it appears certain there's no web3.py issue.

[greese]

The short answer is that I have it working. It requires a cytools build from source.

The long answer:

The Python environment within AWS lambda is 3.6 and built WITHOUT --with-fpectl, but most environments are built with it. To build a proper lambda environment, here are the steps:

1. You need Ubuntu 17.04. It's the only official AMI that can support 3.6. The AWS Linux ones that exist for Lambda do not support 3.6. Go figure. You actually need to upgrade an Ubuntu 16.04 to 17.04. Fun.
2. Do a normal virtualenv configuration of your 3.6 environment.
3. BEFORE pip installing ethereum or web3.py, you need to manually build cytoolz.

git clone https://github.com/pytoolz/cytoolz.git
cd cytoolz
git checkout release
make

4.

for file in VITUALENVROOT/lib/python3.6/site-packages/cytoolz/*; do
  cp -R cytoolz/`basename $file`VIRTUALENVROOT/lib/python3.6/site-packages/cytoolz/;
done

1. pip install ethereum -y
2. pip install web3 -y
3. [if you are deploying new contracts] pip install -y py-solc
4. Package up your lambda

Building cytoolz on your Ubuntu system with figure it all out.

[greese]

Closing this out as it isn't web3.py issue.

[carver]

Great, thanks for posting the writeup here!

[sot528]

Thank you for your post.
And FYI: I could use just pip for ethereum & web3 when I used Amazon Linux docker image for make .zip file.

• https://github.com/sot528/lambda-web3py/blob/master/Dockerfile

[iteratelance]

@greese

I tried this approach, albeit using an older version of Ubuntu 16.04 but was able to get everything to compile and work locally by installing python3.6-dev, build-essential, and libssl-dev. However, unless I'm misreading, doesn't your script assume that cytoolz is already in the virtualenv and copies files that exist in the new build directory to overwrite existing files in the virtualenv?

Did you run the make command within the virtualenv?

I tried mv cytoolz VIRTUALENVROOT/lib/python3.6/site-packages/ but to no avail. Works locally but not on Lambda.

Also the snippet appears to be missing a space between $file and VIRTUALENVROOT unless again I'm in error.

for file in VITUALENVROOT/lib/python3.6/site-packages/cytoolz/*; do
  cp -R cytoolz/`basename $file` VIRTUALENVROOT/lib/python3.6/site-packages/cytoolz/;
done

[iteratelance]

I was able to get this working simply by starting with an Ubuntu 18.04 image and adding the following packages via
sudo apt-get update && sudo apt-get install

        - python3.6-dev
        - build-essential
        - libssl-dev
        - python3-pip
        - locales

Then just ran virtualenv and pip as usually and all was well.