Cookiecutter-django: Problems with Cookiecutter + Docker + PyCharm

[Braintelligence]

Hello everyone!

I have some issues getting comfy with Cookiecutter, Docker and PyCharm.

I'm using current commit ( https://github.com/pydanny/cookiecutter-django/commit/d95bdc6a378062382c536da07364bf8c46f8ea9c ) with latest PyCharm Professional (2018.2.2) and latest docker-ce (18.06.1 ce) and latest docker-compose (1.22.0) on Ubuntu 18.04.1 LTS.

One of my biggest problems is, that the manage.py shell doesn't work as it used to in earlier versions of Cookiecutter. Things like createsuperuser and shell worked flawlessly before.

Now I get errors trying to do this. For example this appears when trying to call on the manage.py shell via CTRL+ALT+R (calling manage.py) and running shell:

docker-compose -f ...PROJECT/local.yml -f .../.PyCharm2018.2/system/tmp/docker-compose.override.362.yml run --rm -p 0.0.0.0:33593:33593 django
Starting janustats_mailhog_1 ... 
Starting janustats_pycharm_helpers_1 ... 
Starting janustats_mailhog_1         ... done
Starting janustats_postgres_1        ... 
Starting janustats_postgres_1        ... done
Starting janustats_pycharm_helpers_1 ... done
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend(['/opt/project', '/opt/.pycharm_helpers/pycharm', '/opt/.pycharm_helpers/pydev'])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run("/home/.../.../PROJECT")
Python 3.6.6 (default, Aug 22 2018, 20:48:31) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 6.5.0
Python 3.6.6 (default, Aug 22 2018, 20:48:31) 
[GCC 6.4.0] on linux
Django 2.0.8
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/environ/environ.py", line 273, in get_value
    value = self.ENVIRON[var]
  File "/usr/local/lib/python3.6/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'DATABASE_URL'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-2c10784a87dc>", line 6, in <module>
    if 'setup' in dir(django): django.setup()
  File "/usr/local/lib/python3.6/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 106, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/project/config/settings/local.py", line 1, in <module>
    from .base import *  # noqa
  File "/opt/project/config/settings/base.py", line 41, in <module>
    'default': env.db('DATABASE_URL'),
  File "/usr/local/lib/python3.6/site-packages/environ/environ.py", line 204, in db_url
    return self.db_url_config(self.get_value(var, default=default), engine=engine)
  File "/usr/local/lib/python3.6/site-packages/environ/environ.py", line 277, in get_value
    raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the DATABASE_URL environment variable

Running migrate and runserver from the predefined run/debug configs works. Copying the migrate config and changing "migrate" to 'createsuperuser' or 'shell' doesn't work. (No debug information to be seen, just an either exiting or seemingly freezing container.)

I've checked the pycharm configuration.rst but couldn't find hints to my problem there.

Please help me to find out how to properly call upon the manage.py shell and how to createsuperuser within PyCharm using docker-compose again. Sorry in advance if I missed something in the documentation regarding my problem.

[Braintelligence]

It seems that there might be a problem with the PyCharm editor. When making migrations I see

janustats_mailhog_1 is up-to-date
Starting janustats_pycharm_helpers_1 ... 
janustats_postgres_1 is up-to-date
Starting janustats_django_1          ... 
Attaching to janustats_django_1
django_1           | PostgreSQL is available

and then nothing.

When running in terminal, I see the question if I have renamed a model to another name and can answer to it.

So I guess it's something PyCharm related, any ideas how I could let PyCharm correctly "hook" into the docker-provided shell?

[sfdye]

Settings -> Project Interpreter -> Add -> Docker-compose -> Choose the local.yml compose file

I believe this is available only for the pro version, not the community version.

[Braintelligence]

Thank you very much @sfdye for pointing this out, I'll try it out when I'm at my PC again. I have the pro version, indeed, so I hope I won't run into problems.

[Braintelligence]

Hey @sfdye , it seems that I already have this setting:

Any other ideas or hints on what I should try/provide? 😦

[sfdye]

How about just from command line:
docker-compose -f local.yml run django python manage.py createsuperuser

[Braintelligence]

As mentioned in https://github.com/pydanny/cookiecutter-django/issues/1766#issuecomment-416590222 running docker-compose commands in terminal (command line) works just fine.
Working with cookiecutter templates within PyCharm and docker worked a few months before, flawlessly. You could execute the manage.py shell within PyCharm and everything.

Either something within PyCharm or cookiecutter must have changed, I believe. I hoped someone could have an idea what we could try.

[sfdye]

What version of PyCharm are you using?

[Braintelligence]

As mentioned in https://github.com/pydanny/cookiecutter-django/issues/1766#issue-354221753 I'm using latest PyCharm Professional (2018.2.2) and latest docker-ce (18.06.1 ce) and latest docker-compose (1.22.0) on Ubuntu 18.04.1 LTS.

[sfdye]

Okay, not sure if it's a PyCharm issue. Can you reproduce this with an older version of PyCharm?

[Braintelligence]

Phew, I'll try it out on my other PC that is running another development environment with an older PyCharm Pro, may take a few days. Will report back!

[sfdye]

I just tried, "Run manage.py Task" works fine for me but "Python Console" failed with the same error. Running manage.py shell from command line works fine too. So I am afraid this is now a PyCharm issue.

[elcolie]

@Braintelligence https://github.com/pydanny/cookiecutter-django/issues/1033
You might need to config it explicitly in the config file

[hangtwenty]

I love PyCharm - but why not close this issue. It's generally a better practice to use a dedicated terminal emulator (I'm partial to Kitty). You're stunting your productivity if you're staying in an IDE-embedded shell 😬

[amcorreia]

You're stunting your productivity if you're staying in an IDE-embedded shell 😬

To run "runserver" I agree. However, it's simpler to just click on green arrow on side of function and say "Run 'Unittests for function X'" than type a long text in terminal.

BTW: I'm having the same problem to run tests.

[DanielFLopez]

I had the same problem and I found a way to partially solve this problem by manually adding the environment variable DATABASE_URL

[browniebroke]

If someone feels like contributing, I'll be happy to help people who want to tackle this.

I suspect the changes should be done inside the .idea directory, it's most likely because our config was generated for a version of Pycharm from a few years ago.

How to get started? First thing is to see what's wrong in our config, we can do it by following the steps below (I think):

1. Generating a dummy project with use_pycharm=y
2. Remove the .idea entries from the dummy project .gitignore:
   https://github.com/pydanny/cookiecutter-django/blob/b039e241302deeeec211ea5bf80c61254dede022/%7B%7Bcookiecutter.project_slug%7D%7D/.gitignore#L174-L195
3. Git add and commit everything
4. Open the project in PyCharm, fix all the config
5. Do a git diff to see what's changed, or better, commit and push a public repo that can be shared here

Even without sending a pull request, just doing the above steps would be super helpful.

[foarsitter]

The compose/production/django/entrypoint scripts adds the DATABASE_URL and CELERY_BROKER_URL env variables. Pycharm does not use the entrypoint script and therefore the variables are missing.

Add the next code after the import statements in File | Settings | Build, Execution, Deployment | Console | Django Console fixes the issue.

import os
os.environ.setdefault("DATABASE_URL","postgres://{}:{}@{}:{}/{}".format(
    os.environ['POSTGRES_USER'], 
    os.environ['POSTGRES_PASSWORD'], 
    os.environ['POSTGRES_HOST'], 
    os.environ['POSTGRES_PORT'], 
    os.environ['POSTGRES_DB']
))
os.environ.setdefault("CELERY_BROKER_URL", os.environ['REDIS_URL'])

Pycharm stores this setting in .idea/workspace.xml. Adding this file will solve the issue I hope.

[Braintelligence]

@browniebroke Should the above commit also be merged?
Nevermind it was just a copypaste of the available merged fix.