Serverless-python-requirements: Corrupted _psycopg shared lib when building on Linux

[brettdh]

I'm using this plugin to deploy a python package with psycopg2, which has a native component. I have dockerizePip: true, and when I build on my MacBook Pro, everything seems fine (and runs fine in AWS Lambda):

➜ file samm-mine/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so                                                                                                                                                                    
samm-mine/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared
object, x86-64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=4259d90be15aac6fb0a16833bbf9c767d9df2f57, not stripped

But when my teammate runs the same build on Ubuntu 16.04, the _psycopg shared lib is corrupted:

➜ file samm-his/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so                                                                                                                                                                     
samm-his/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared
object, x86-64, version 1 (SYSV), corrupted program header size, corrupted section header size

and psycopg2 throws a traceback:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module:
/var/task/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so: ELF file's phentsize
not the expected size

My OS:

➜ uname -a
Darwin me 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017;
root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64

His OS:
```
$ uname -a
Linux him 4.10.0-33-generic #37~16.04.1-Ubuntu SMP Fri Aug 11 14:07:24 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux

[dschep]

Are the corrupted file errors from Lambda during invocation or something during the packaging/deployment? I assume your teammate is also using the dockerizePip: true option, correct?

[brettdh]

Are the corrupted file errors from Lambda during invocation or something during the packaging/deployment?

Both. The file command above shows corruption, and the lambda invocation also does.

Interesting: when he runs sis requirements install, the .so is NOT corrupted:

$ file .requirements/psycopg2/_psycopg*
.requirements/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so: ELF 64-bit LSB shared object,
x86-64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=4259d90be15aac6fb0a16833bbf9c767d9df2f57, not stripped

I assume your teammate is also using the dockerizePip: true option, correct?

Yep; we have the same serverless config, same requirements.txt, etc.

[brettdh]

Clarification: when I posted the corrupted .so file info, I had downloaded the .zip from his deployment bucket and extracted it on my mac.

[brettdh]

The plot thickens! In the zip file inside .serverless, the .so is corrupted.

[dschep]

Thanks for the details. I don't have any ideas as to what's wrong, but I also use linux so hopefully I'll be able to replicate it.

[alexdebrie]

@brettdh Do you know what version of Serverless your friend is using? The 1.21 version introduced a regression around zipping binary packages due to an encoding issue -- see issue here. There's a PR open to fix it, but switching back to 1.20.2 would fix for the time being.

[brettdh]

Just tested this out. I can confirm that 1.21 does indeed cause the problem above, and that rolling back to 1.20.2 fixes it. Thanks!

Closing, as it's not a problem with this plugin.

[dschep]

Thanks for the heads up about the zipping issue in serverless @alexdebrie!

[alexdebrie]

@brettdh @dschep v1.21.1 of the Framework released today with a fix for the binary encoding issues. Feel free to ping me if this doesn't solve your issue.