Cookiecutter-django: Set the DATABASE_URL environment variable in GitHub CI

[RamsesMartinez]

This step throws the error in a totally new project

Run Django Tests
docker-compose -f local.yml exec -T django pytest

But if I run the command on my machine

docker-compose -f local.yml run --rm django pytest

It works without problems, why does the default file use exec? and why does that command throw the error of not finding the variable?

$ docker-compose -f local.yml exec django pytest
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 273, in get_value
    value = self.ENVIRON[var]
  File "/usr/local/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'DATABASE_URL'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 180, in console_main
    code = main()
  File "/usr/local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 136, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 313, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.8/site-packages/_pytest/helpconfig.py", line 99, in pytest_cmdline_parse
    config = outcome.get_result()  # type: Config
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 932, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1204, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1107, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.8/site-packages/pytest_django/plugin.py", line 323, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 76, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 63, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 142, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/config/settings/test.py", line 5, in <module>
    from .base import *  # noqa
  File "/app/config/settings/base.py", line 43, in <module>
    DATABASES = {"default": env.db("DATABASE_URL")}
  File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 204, in db_url
    return self.db_url_config(self.get_value(var, default=default), engine=engine)
  File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 277, in get_value
    raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the DATABASE_URL environment variable

It works fine

$ docker-compose -f local.yml run --rm django pytest
Starting ecommerce_postgres ... done
PostgreSQL is availablehog  ... done
Test session starts (platform: linux, Python 3.8.5, pytest 6.0.2, pytest-sugar 0.9.4)
django: settings: config.settings.test (from option)
rootdir: /app, configfile: pytest.ini
plugins: celery-4.4.6, django-3.10.0, Faker-4.1.3, sugar-0.9.4
collecting ... 
 ecommerce_fyg/users/tests/test_drf_urls.py ✓✓✓                                                                                                                           18% █▊        
 ecommerce_fyg/users/tests/test_drf_views.py ✓✓                                                                                                                           29% ██▉       
 ecommerce_fyg/users/tests/test_forms.py ✓                                                                                                                                35% ███▌      
 ecommerce_fyg/users/tests/test_models.py ✓                                                                                                                               41% ████▎     
 ecommerce_fyg/users/tests/test_tasks.py ✓                                                                                                                                47% ████▊     
 ecommerce_fyg/users/tests/test_urls.py ✓✓✓                                                                                                                               65% ██████▌   
 ecommerce_fyg/users/tests/test_views.py ✓✓✓✓✓✓                                                                                                                          100% ██████████

Results (0.94s):

In another answer I saw that they suggest to previously configure the variable in the terminal, but then, how should I configure the ci.yml github file?

[arnav13081994]

@RamsesMartinez

You exposed a very interesting issue with the current setup.

First of all, the difference between exec and run commands is that the latter starts up a new container to run a command while the former runs the command in a running container.

It works without problems, why does the default file use exec?

To answer your first question, it makes more sense, in my opinion, to run the test command in a running django container than to spin up a new django container just to run the test commands. Please note that run will also start any other services this service depends on, which is not the case here since docker-compose up has already been executed and services like redis and postgres are already up and running.

Moreover, in a real-world scenario, user requests/commands/inputs would be executed in a running container and a new container will not be spun up for every user request/command/input, so matching that with the testing environment should be the aim, in my opinion. I know containers are ephemeral in nature but that is from the perspective of data persistence and not command execution, again my opinion.

Now this seemingly small distinction between run and exec commands has interesting consequences for bash variables.

why does that command throw the error of not finding the variable?

It's the Entrypoint script at ./compose/production/django/entrypoint that is exporting the DATABASE_URL to the Root user's session for the django service. The Entrypoint script is run only at container creation time at which point all the required environment variables are created and exported. Please also note that whenever a container is restarted, it is actually destroyed and a new container is spun up. I think of containers simply as processes running on the docker host.

So when you used run to spin up a new container, the bash variables got created and exported to the root user's session and then the test command got executed in the same session after which the container exited. But when exec was used, the command was still run in the root user's session except that this was a new session and hence had no access to the same bash variables! By default, bash variables do not persist across sessions.

Now, one could try to get around this issue, but the only way I am aware of is by saving the variables in a file. But then that would be the same thing as creating another entry in the relevant .env file.

I would suggest that instead of using run to run pytest you could simply do something like this in the ci.yml file:

name: CI

# Enable Buildkit and let compose use it to speed up image building
env:
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1
  # create a top-level GitHub action env key that will be accessible to every GitHub action step and job
  DATABASE_URL: postgres://.......

......

      - name: Run Django Tests (passing DATABASE_URL as an env var to exec)
        run:  docker-compose -f local.yml exec -e DATABASE_URL=${DATABASE_URL} -T django pytest

......

Or create a new entry in the .env file and use that in your settings file. I personally prefer the latter approach except that I would use docker secrets for that

@browniebroke I suspect this kind of issue hasn't been faced before because all tests in this project are being executed with the run command. I am not sure if the decision to use run instead of exec was deliberate. I think we should use exec throughout the project, but I understand if that would entail a lot of changes and may simply not be worth the effort. Please let me know and I would also update the tests for Github Actions and raise a PR.

Or perhaps this might be a good time to switch to docker secrets and get rid of such issues altogether. Compose and swarm both support secrets and I'm also using them personally. Let me know.

[Andrew-Chen-Wang]

@arnav13081994 I personally like run mostly because Elasticsearch takes forever to launch and my computer can't handle a Docker orchestration being up for more than 10 minutes without dying... Yeah for test driven code...

I'd be interested in seeing a docker secrets implementation; I've personally never used it, but at a first glance at the docs, I just see the .env files being much easier to work with. I can fidget around with all the variables I need all in 3 ish files.

@RamsesMartinez I filed a PR here awhile ago to just erase Docker from our CIs since they took... 4-5 minutes? Ah #2637 cut down to 1.5 minutes. You can see a Travis implementation here with celery and stuff. That's the other thing I like about .env files: you can share those local environment variables between your team members in case you start adding more features.

I'm ngl, I highly recommend that approach over using the Travis and Gitlab CI implementation that I first added for Docker.

Because when Arnav says:

I am not sure if the decision to use run instead of exec was deliberate

It was because I had only used Docker for... 2 weeks? and only learned programming for... 6 weeks? Hahaha so a lot of stuff that I contributed to here was me fidgeting around with code. If anything, I still don't think Docker should be the way to our CIs, at least not for the Django/Python/Javascript apps themselves since you want to take advantage of the CI cache. Edit: this was the PR if you were curious as to why I implemented the CIs to be like that for Docker: #2539