Serverless-python-requirements: Django and mysqlclient

[brett-matthews]

Im attempting to use serverless-python-requirements + serverless-wsgi with Django using mysql database.

Ive read over the README numerous times but haven't been able to get this to work.

Ive created a repo as an example.
https://github.com/brett-matthews/serverless-django-wsgi-requirements-test

I would be happy to contribute to the README here if we could manage to solve this which may help other users in the future.

EDIT - NOW WORKING!!

EDIT

CloudWatch Logs from the Lambda

File "/var/task/django/db/backends/mysql/base.py", line 15, in <module>
import MySQLdb as Database
File "/var/task/MySQLdb/__init__.py", line 18, in <module>
from . import _mysql
ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/task/wsgi_handler.py", line 44, in import_app
wsgi_module = importlib.import_module(wsgi_fqn_parts[-1])
File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/task/project/wsgi.py", line 16, in <module>
application = get_wsgi_application()
File "/var/task/django/core/wsgi.py", line 12, in get_wsgi_application
django.setup(set_prefix=False)
File "/var/task/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/var/task/django/apps/registry.py", line 112, in populate
app_config.import_models()
File "/var/task/django/apps/config.py", line 198, in import_models
self.models_module = import_module(models_module_name)
File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/var/task/django/contrib/auth/models.py", line 2, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "/var/task/django/contrib/auth/base_user.py", line 47, in <module>
class AbstractBaseUser(models.Model):
File "/var/task/django/db/models/base.py", line 101, in __new__
new_class.add_to_class('_meta', Options(meta, app_label))
File "/var/task/django/db/models/base.py", line 305, in add_to_class
value.contribute_to_class(cls, name)
File "/var/task/django/db/models/options.py", line 203, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/var/task/django/db/__init__.py", line 33, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/var/task/django/db/utils.py", line 202, in __getitem__
backend = load_backend(db['ENGINE'])
File "/var/task/django/db/utils.py", line 110, in load_backend
return import_module('%s.base' % backend_name)
File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/var/task/django/db/backends/mysql/base.py", line 20, in <module>
) from err
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
Did you install mysqlclient?
START RequestId: 3a38fcec-4dd5-46c3-926d-75b791a980f1 Version: $LATEST
module initialization error: Unable to import project.wsgi.application
END RequestId: 3a38fcec-4dd5-46c3-926d-75b791a980f1

[miketheman]

@brett-matthews can you be more specific about what isn't working? Thanks for providing the repo link, but any error messages, console output are going to provide more context.

[miketheman]

The only error I've received when running sls package was a notice about the libmysql not being found, so I swapped the commented lines and was able to package.
https://github.com/brett-matthews/serverless-django-wsgi-requirements-test/blob/ccd5526a860da8097ed2bd9a585fa8e975f6eb17/serverless.yml#L10-L11

Are you getting a runtime error? Is this the same as #317 ?

[brett-matthews]

My bad I should of attached the Error @miketheman.. Added to original post.

Let me try again re:
https://github.com/brett-matthews/serverless-django-wsgi-requirements-test/blob/ccd5526a860da8097ed2bd9a585fa8e975f6eb17/serverless.yml#L10-L11

I left the commented out lines to demonstrate I tried both.. Or so I thought i did :)

Thanks for your help!

[brett-matthews]

Still receive this error (from original post)

ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory

Regardless of either of these options in dockerExtraFiles; tried --force on deploy too

      - /usr/lib64/mysql57/libmysqlclient.so.1020
      - /usr/lib64/mysql/libmysqlclient.so.18

[brett-matthews]

So I set these values in my serverless.yml file and now it is working!

  pythonRequirements:
    useDownloadCache: false
    useStaticCache: false

Im not exactly sure why, perhaps I had a cached build that was broken?

[littlebtc]

So I set these values in my serverless.yml file and now it is working!

  pythonRequirements:
    useDownloadCache: false
    useStaticCache: false

Im not exactly sure why, perhaps I had a cached build that was broken?

I had ran into similar issues, it seems that the package inclusion/exclusion process will not re-run if you have static cache.

Remove the static cache or invalidate it by changing requirements.txt should re-package correctly.

As you had successfully built it, it should be safe to enable caches again.

[miketheman]

@brett-matthews Glad to hear it's working! I've tagged this to leave open for now.

I know this might be hard to confirm - in the README we note that you should try and confirm that your file was included in the zip package:
https://github.com/UnitedIncome/serverless-python-requirements/blob/0738379e04d43a120a800a66b40f97eb02a100d8/README.md#native-code-dependencies-during-runtime

Is that something you tried while debugging this?

Going to leave this open for others to see for now, at least until we learn more about the behaviors and consider what might be a good tradeoff between automatically clearing caches for extra files, or some other approach.

[brett-matthews]

As in looking to see what files were included in the Docker image?

https://github.com/brett-matthews/serverless-django-wsgi-requirements-test/blob/ccd5526a860da8097ed2bd9a585fa8e975f6eb17/serverless.yml#L10-L11

For someone not familiar with Docker such as myself this took a bit of research to try this file;

- /usr/lib64/mysql/libmysqlclient.so.18

which was different to the tutorial. With the caching trouble I experienced it was even more confusing.

Not sure on the direction you could take? Maybe a simple note in the Native Code Dependencies During Runtime section of the README to turn off caching initially until happy with a working build?

[kahlil29]

@brett-matthews Thanks for opening this issue, providing so many details and leaving it open. Helped me after I hit this issue when trying to use a Python resizing Lambda with Pillow-SIMD dependency.
Struggled for a while before coming across this issue and reading it helped my understanding of the issue and feature.

[zees-dev]

@brett-matthews Thanks from me too.
Was trying to run Django 3.1 serverlessly - hit roadblock once i found out i couldn't connect to remote MySQL DB due to a mysqlclient error (similar to yours).
I encountered the following error (after a long stack trace): NameError: name '_mysql' is not defined
Have successfully resolved the issue now (thanks to the fix you've provided) 👍

[zees-dev]

For anyone else who would like to get Django 3.1 (latest at this time) running on serverless infrastructure while also being able to connect to MySQL database; i have setup a repo for this:
https://github.com/zees-dev/serverless-django