Pyinstaller: Django hook ignores DJANGO_SETTINGS_MODULE environment variable

[sean0x42]

Is your feature request related to a problem? Please describe.

Django allows users to set the value of the DJANGO_SETTINGS_MODULE environment variable so that multiple settings modules can be created, and a specific one can be chosen depending on the environment.

mysite/
|-- mysite/
|   |-- __init__.py
|   |-- settings/
|   |   |-- __init__.py
|   |   |-- base.py
|   |   |-- ci.py
|   |   |-- development.py
|   |   |-- production.py
|   |   +-- staging.py
|   |-- urls.py
|   +-- wsgi.py
+-- manage.py

Pyinstaller ignores this variable when building, and assumes that all settings modules must be found at mysite/settings.py, or mysite/settings/. You end up getting several error messages like the following:

AttributeError: Module 'mysite.settings' has no attribute 'INSTALLED_APPS'

and

django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

Describe the solution you'd like
For the django hook to use os.environ.setdefault, such that the default settings module is mysite.settings, but users may configure it to be something else.

Describe alternatives you've considered
None

Additional context

It looks like there are a number of locations in the code base that this assumption is made, so I've collected some examples below:

• https://github.com/pyinstaller/pyinstaller/blob/03d42a2a25269f453a81f066d4b34a5dd330a66c/PyInstaller/utils/hooks/django.py#L31
• https://github.com/pyinstaller/pyinstaller/blob/03d42a2a25269f453a81f066d4b34a5dd330a66c/PyInstaller/utils/hooks/django.py#L61
• https://github.com/pyinstaller/pyinstaller/blob/03d42a2a25269f453a81f066d4b34a5dd330a66c/PyInstaller/utils/hooks/django.py#L68
• https://github.com/pyinstaller/pyinstaller/blob/03d42a2a25269f453a81f066d4b34a5dd330a66c/PyInstaller/hooks/hook-django.py#L46
• https://github.com/pyinstaller/pyinstaller/blob/03d42a2a25269f453a81f066d4b34a5dd330a66c/PyInstaller/hooks/hook-django.py#L70

The pattern of splitting settings modules up into different files was introduced to me in this article: https://simpleisbetterthancomplex.com/tips/2017/07/03/django-tip-20-working-with-multiple-settings-modules.html

[Legorooj]

So... You want PyInstaller to save the value of DJANGO_SETTINGS_MODULE and load it at runtime so that Django loads the correct module - eg DJA..ULE=mysite.settings.production is set by default at runtime?

[sean0x42]

Well I was actually just thinking at build time, that way the relevant config file would be included. But now that I think of it, you're right it would also have to be stored for runtime. Is that possible, or am I misunderstanding the architecture of pyinstaller?

[sean0x42]

@Legorooj I thought about this overnight, and I remembered that manage.py actually sets a default value for DJANGO_SETTINGS_MODULE. It's actually even the very first thing it does!

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""

import os
import sys

def main():
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
    # ...


if __name__ == "__main__":
    main()

What this means, is that pyinstaller wouldn't necessarily need to store the value of the environment variable for runtime, since Django can and does already set a value at runtime. Pyinstaller simply needs to read the variable at build time, to include the correct settings module.

[Legorooj]

@sean0x42 hang on. So manage.py would/could have the correct settings module linked in the setdefault statement, right? That means that before building, you could just change it (the default value) to production, before building, right? That means that PyInstaller just needs to include the correct settings module?

[sean0x42]

@sean0x42 hang on. So manage.py would/could have the correct settings module linked in the setdefault statement, right? That means that before building, you could just change it (the default value) to production, before building, right? That means that PyInstaller just needs to include the correct settings module?

Yes that's 100% correct.

The only downside with this approach, is that since it sets a default value, theoretically the user running the final .exe could have an environment variable set in their operating system that is different to the default. To prevent that edge case, the following line:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings.production")

should be replaced with this:

os.environ["DJANGO_SETTINGS_MODULE"] = "mysite.settings.production"

Which could be documented in the wiki, and could simply be pre-requisite if you wish to use multiple settings files with pyinstaller.

[Legorooj]

So, that can be documented in the wiki. Now, if you make those edits, do you get an error?

[sean0x42]

So, that can be documented in the wiki. Now, if you make those edits, do you get an error?

Just gave it a shot, and it works just fine.

[Legorooj]

Ok, I'll add some documentation to the wiki. I assume you hard-set the environment variable, or did you soft-set it (setdefault)?

[sean0x42]

Ok, I'll add some documentation to the wiki. I assume you hard-set the environment variable, or did you soft-set it (setdefault)?

Hard set, as in os.environ["DJANGO_SETTINGS_MODULE"] = "mysite.settings.production"

Sorry, just to be clear, making that change doesn't fix the overall issue here. It just means that pyinstaller doesn't need to do anything at runtime. Would still need to update pyinstaller to respect the value of DJANGO_SETTINGS_MODULE at build time. So solving the overall issue would require both documentation and some changes to the django hook.

[Legorooj]

Ok, I'm confused. Didn't you just say it worked fine?

[sean0x42]

Ok, I'm confused. Didn't you just say it worked fine?

Yea sorry for the confusion, I thought you meant: does making that change cause any additional errors with Django. Will try to be clearer in future 👍

[Legorooj]

@sean0x42 Ok what I meant was when building, with this configuration, what are the errors?

[sean0x42]

@sean0x42 Ok what I meant was when building, with this configuration, what are the errors?

Same as described in the original post. Making that change doesn't change anything about the pyinstaller build, since pyinstaller totally ignores the DJANGO_SETTINGS_MODULE environment variable.

AttributeError: Module 'mysite.settings' has no attribute 'INSTALLED_APPS'

and

django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

[Legorooj]

Which version of Django are you using?

[sean0x42]

2.2. Would you like me to try with version 3?

I actually didn't realise there was a new major version released...

[Legorooj]

No thanks. Can you try with 2.1.8? That's the last version the current hook is confirmed to work with - 2.2+ will need a hook update.

[sean0x42]

So I downgraded to Django 2.1.8, and then re-ran. I'm also using pipenv if that matters. DJANGO_SETTINGS_MODULE is set to [redacted].settings.production.

Here's the entire output:

$ pyinstaller --name [redacted] --onefile manage.py
141 INFO: PyInstaller: 4.0.dev0+03d42a2a25
141 INFO: Python: 3.7.2
142 INFO: Platform: Windows-10-10.0.18362-SP0
143 INFO: wrote C:\Users\sean0x42\workspace\[redacted]\server\[redacted].spec
148 INFO: UPX is not available.
156 INFO: Extending PYTHONPATH with paths
['C:\\Users\\sean0x42\\workspace\\[redacted]\\server',
 'C:\\Users\\sean0x42\\workspace\\[redacted]\\server']
165 INFO: checking Analysis
265 INFO: Building because C:\Users\sean0x42\workspace\[redacted]\server\manage.py changed
266 INFO: Initializing module dependency graph...
271 INFO: Caching module graph hooks...
285 INFO: Analyzing base_library.zip ...
5119 INFO: Caching module dependency graph...
5274 INFO: running Analysis Analysis-00.toc
5286 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by c:\users\sean0x42\.virtualenvs\server--spyqgjc\scripts\python.exe
5825 INFO: Analyzing C:\Users\sean0x42\workspace\[redacted]\server\manage.py
5901 INFO: Processing pre-find module path hook distutils from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
5902 INFO: distutils: retargeting to non-venv dir 'C:\\Users\\sean0x42\\AppData\\Local\\Programs\\Python\\Python37\\Lib'
7090 INFO: Processing pre-find module path hook site from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-site.py'.
7091 INFO: site: retargeting to fake-dir 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\fake-modules'
13286 INFO: Processing module hooks...
13286 INFO: Loading module hook 'hook-distutils.py' from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks'...
13289 INFO: Loading module hook 'hook-django.core.cache.py' from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks'...
13483 INFO: Loading module hook 'hook-django.core.mail.py' from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks'...
13576 INFO: Loading module hook 'hook-django.core.management.py' from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks'...
13605 INFO: Import to be excluded not found: 'matplotlib'
13605 INFO: Import to be excluded not found: 'IPython'
13605 INFO: Import to be excluded not found: 'tkinter'
13605 INFO: Loading module hook 'hook-django.db.backends.py' from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks'...
14779 WARNING: Hidden import "django.db.backends.__pycache__.base" not found!
14779 INFO: Loading module hook 'hook-django.py' from 'c:\\users\\sean0x42\\.virtualenvs\\server--spyqgjc\\lib\\site-packages\\PyInstaller\\hooks'...
Traceback (most recent call last):
  File "<string>", line 41, in <module>
  File "<string>", line 36, in walk_packages
  File "<string>", line 36, in walk_packages
  File "<string>", line 20, in walk_packages
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\admin\__init__.py", line 5, in <module>
    from django.contrib.gis.admin.options import GeoModelAdmin, OSMGeoAdmin
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\admin\options.py", line 2, in <module>
    from django.contrib.gis.admin.widgets import OpenLayersWidget
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\admin\widgets.py", line 3, in <module>
    from django.contrib.gis.gdal import GDALException
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\gdal\__init__.py", line 28, in <module>
    from django.contrib.gis.gdal.datasource import DataSource
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\gdal\datasource.py", line 39, in <module>
    from django.contrib.gis.gdal.driver import Driver
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\gdal\driver.py", line 5, in <module>
    from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\gdal\prototypes\ds.py", line 9, in <module>
    from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\contrib\gis\gdal\libgdal.py", line 43, in <module>
    % '", "'.join(lib_names)
django.core.exceptions.ImproperlyConfigured: Could not find the GDAL library (tried "gdal202", "gdal201", "gdal20", "gdal111", "gdal110", "gdal19"). Is GDAL installed? If it is, try setting GDAL_LIBRARY_PATH in your settings.
17885 INFO: Determining a mapping of distributions to packages...
34004 INFO: Packages required by django:
['pytz']
34005 INFO: Django root directory C:\Users\sean0x42\workspace\[redacted]\server\server
Traceback (most recent call last):
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\utils\hooks\subproc\django_import_finder.py", line 29, in <module>
    django.setup()
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\conf\__init__.py", line 57, in __getattr__
    self._setup(name)
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\conf\__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\django\conf\__init__.py", line 126, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
34613 INFO: Collecting Django migration scripts.
Traceback (most recent call last):
  File "C:\Users\sean0x42\AppData\Local\Programs\Python\Python37\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\sean0x42\AppData\Local\Programs\Python\Python37\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\sean0x42\.virtualenvs\server--SPYQGJc\Scripts\pyinstaller.exe\__main__.py", line 9, in <module>
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\building\build_main.py", line 735, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\building\build_main.py", line 682, in build
    exec(code, spec_namespace)
  File "C:\Users\sean0x42\workspace\[redacted]\server\[redacted].spec", line 17, in <module>
    noarchive=False)
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\building\build_main.py", line 247, in __init__
    self.__postinit__()
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\building\datastruct.py", line 160, in __postinit__
    self.assemble()
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\building\build_main.py", line 424, in assemble
    self.graph.process_post_graph_hooks()
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\depend\analysis.py", line 359, in process_post_graph_hooks
    module_hook.post_graph()
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\depend\imphook.py", line 419, in post_graph
    self._load_hook_module()
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\depend\imphook.py", line 386, in _load_hook_module
    self.hook_module_name, self.hook_filename)
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\compat.py", line 611, in importlib_load_source
    return mod_loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 907, in load_module
  File "<frozen importlib._bootstrap_external>", line 732, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\hooks\hook-django.py", line 70, in <module>
    installed_apps = eval(get_module_attribute(package_name + '.settings', 'INSTALLED_APPS'))
  File "c:\users\sean0x42\.virtualenvs\server--spyqgjc\lib\site-packages\PyInstaller\utils\hooks\__init__.py", line 284, in get_module_attribute
    'Module %r has no attribute %r' % (module_name, attr_name))
AttributeError: Module 'server.settings' has no attribute 'INSTALLED_APPS'

[Legorooj]

How about, instead of SETTINGS_MOD=mysite.setttings.production, SETTINGS_MOD=mysite.settings and settings.__init__.py simply contains from .production import *?

[sean0x42]

Okay, so it seems to build successfully after making that change. However now I'm faced with an unrelated issue, in that when I run the .exe the program just hangs. And then after a very long time if I ctrl+c, I get the following exception:

ModuleNotFoundError: No module named 'django_extensions'

Most likely the solution you proposed fixes the problem, and this is some other issue, but I'm not 100% certain. Thoughts?

Edit: To be clear, I'm using another package called django_extensions. That hasn't just come out of the blue.

[Legorooj]

@sean0x42 https://github.com/django-extensions/django-extensions?

[sean0x42]

@sean0x42 https://github.com/django-extensions/django-extensions?

Yep, that's the one! I think the only place it is actually referenced is in INSTALLED_APPS, so it makes sense that pyinstaller would miss it.

[Legorooj]

Hmm. You'll need a collect_all hook for django_extensions. And make sure to add django_extensions to hiddenimports with --hidden-import=django_extensions.

[sean0x42]

Hmm. You'll need a collect_all hook for django_extensions. And make sure to add django_extensions to hiddenimports with --hidden-import=django_extensions.

Okay thanks heaps @Legorooj. Will look more into it.

Otherwise it seems like adding from production import * to __init__.py resolves this point.

[Legorooj]

@sean0x42 :smile:. Once you've written the hook for django_extensions, let me know if the issues still persist.

[sean0x42]

@Legorooj where should I create the hook, in the pyinstaller repo, or in the django_extensions repo?

Edit: Or maybe I should just create one in my repo for now, and offer it to some other project via a PR later.

[Legorooj]

Test it with --additional-hooks-dir=hooks and have the hook-django_extensions.py inside said hooks subfolder. We can talk integration later, we need to check it works first.

[sean0x42]

@Legorooj Have been playing around (so far unsuccessfully) with creating a hook for django_extensions, and I noticed something weird. When building to a directory rather than a file, I noticed that only migrations are being included in the final output, and no actual application logic.

This is all that gets included for the support app:

And here's how much there should be:

Is this normal? I am using Django REST framework, which introduces an alternative routing system. If I had to guess, pyinstaller doesn't support this and so doesn't know how to find my views and logic.

[Legorooj]

@sean0x42 the Django hook needs a major upgrade. It's on my to-do list. After rewriting modulegraph, redoing unicode support, figuring out python 3.8 support, and removing hooks from PyInstaller into a sister package. I've got a bit to do.

Try the following:

- manage.py
- hooks
  - hook-django_extensions.py

And inside the hook:

from PyInstaller.utils.hooks import collect_all

datas, binaries, hiddenimports = collect_all('django_extensions')

And build with --additional-hooks-dir=hooks. Make sure that you pass the correct path to hooks if you're running from another directory.

[sean0x42]

Yea sounds like you've got heaps to do! Thanks for all your support with a workload like that.

I actually did exactly what you outlined in your comment, line for line, character for character. Yet it didn't seem to actually run the hook I created. Like I think it found the hook, but then never found the django_extensions import or something and so never ran the hook. Either way, my Django app is far from usual in many ways, and I think that may be contributing to the problems I'm facing.

I actually just resigned from my job believe it or not haha, so I'm going to spend the next few weeks focusing on cleaning that up. Unfortunately that means I don't have much time to finish things up here. I'm looking to get more into some open source projects when I do have time though, so maybe I can come back then and offer you some help with everything.

For now, to help resolve this issue, would you like me to edit the wiki to include the work around we discussed here for handling multiple settings files? I'd be more than happy to write that up.

[Legorooj]

Put a raise RuntimeError in the hook. The build will crash if the hook is being run.

For now, to help resolve this issue, would you like me to edit the wiki to include the work around we discussed here for handling multiple settings files? I'd be more than happy to write that up.

Please! Well, assuming you can edit it - I'm not the project admin. If you can't, write up the markdown in a comment and I'll copy it in.

I actually just resigned from my job believe it or not haha, so I'm going to spend the next few weeks focusing on cleaning that up. Unfortunately that means I don't have much time to finish things up here. I'm looking to get more into some open source projects when I do have time though, so maybe I can come back then and offer you some help with everything

Help with the Django hooks would be great! They need a lot of work. (And anything else of course ;-))

Note: For Django, we aren't moving it's hooks out of PyInstaller like most hooks. Major hooks are staying built it. Django classifies as one, alongside matplotlib, PyQt5, etc.

[Legorooj]

Actually, I'm going to close this issue; the original problem is resolved, we just need to add the correct notes in the wiki.

[sean0x42]

Okay, I've edited the wiki. You may want to just read over my changes and make sure you're happy with it.

https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Executable-From-Django

[Legorooj]

@sean0x42 looks good! If you do open a PR to fix the Django hooks - or you want to discuss it, in which case open a new issue (feature template, called something like "Roadmap to update django hooks") - tag me.

[lstolcman]

Hi, I also use "multiple" settings in my project, like an article in the first post describes.

Finally, I had to migrate to pyinstaller 4.0, so I cleaned up my hooks - diff looks good and readable.
Link to the branch: https://github.com/lstolcman/pyinstaller/tree/multiple_django_settings

How does it work: set env variable DJANGO_SETTINGS_MODULE - overrides default settings file path to your own.

Changes:

• settings - default or overriden by variable
• find_url_callbacks in django_import_finder.py was removed - has no impact in case of compiling my app, don't know whether it is needed in django 2.2+ or not.
• All files from directory are excluded (datas += collect_data_files(package_name, excludes=os.listdir())) - due to changed behavior of collect_data_files. Without this, I got all data copied into the result project (including development files, .git folder, etc.)

@Legorooj thoughts?

[Legorooj]

@lstolcman are you planning/would you be able to submit this in a PR? I can review and discuss much more easily there.