Cookiecutter-django: docker/celery exits because celery is run as root

[pleasantone]

Note: for support questions, please use the cookiecutter-django tag on stackoverflow. This repository's issues are reserved for feature requests and bug reports. If you need quick professional paid support for your project, contact [email protected].

• *I'm submitting a ... *

  • [x] bug report
  • [ ] feature request
  • [ ] support request => Please do not submit support request here, see note at the top of this template.

• Do you want to request a feature or report a bug?

Bug

• What is the current behavior?

Celery is running as root in its docker container, and is configured to accept messages serialized with pickle. Its automatic checks are causing it to exit.

celeryworker_1  | Running a worker with superuser privileges when the
celeryworker_1  | worker accepts messages serialized with pickle is a very bad idea!
celeryworker_1  |
celeryworker_1  | If you really want to continue then you have to set the C_FORCE_ROOT
celeryworker_1  | environment variable (but please think about this before you do).
celeryworker_1  |
celeryworker_1  | User information: uid=0 euid=0 gid=0 egid=0
celeryworker_1  |
xxxx_celeryworker_1 exited with code 1

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

base cookiecutter install with both docker and celery enabled

• What is the expected behavior?

celery should run, not terminate

• What is the motivation / use case for changing the behavior?

task queues

• Please tell us about your environment:

github master of cookiecutter-django as of 8/27/2017

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

Two solutions, both of them are reasonably docker-friendly.

1. Celery could be run as a non-root user (e.g. as 'app' or 'daemon')
2. Add the C_FORCE_ROOT environment variable to the celery container.

I -personally- believe that even in containers, apps should not run as root, as its a good security practice, but I know others think I'm a luddite. I've spent plenty of years fixing security mistakes.

[jayfk]

The reason for running the Django container as root is that there were permission problems with boot2docker on macOS and Windows. Has this changed with Docker for Mac/Windows?

The main problem is that the custom user running the container needs r/w permissions on the mounted app directory.

[pleasantone]

If I recall correctly, django runs just fine as django in the django container.
We're talking about the celery_worker container. I can verify that django works great in docker on a mac as not-root.

[jayfk]

We're talking about the celery_worker container.

Sure, that was just some historical context on why the container runs as root.

[cjvanderlinden]

Is there any update or recommendation on what should be done for these cases as it stands?
(specifically from the project owners)

Since everything else seems to run as root, I am guessing the quick fix is to just use the C_FORCE_ROOT

EDIT: I may be missing something really easy, but adding this to my config/settings/base.py file as either:

C_FORCE_ROOT = 'true' 
C_FORCE_ROOT = True

doesn't seem to work

[mjj55409]

It's only running locally where it's a problem. In my dev.yml I added C_FORCE_ROOT to the environment for the celery worker:

 celeryworker:
    environment:
      - C_FORCE_ROOT=true
    depends_on:
      - redis
      - postgres
      - mailhog
    ports: []
    command: /start-celeryworker-dev.sh

[cjvanderlinden]

Thank you!

Still getting into that docker mindset.

Would you be able to just go a bit more in depth on why its only an issue with the local.yml and not production.yml

I haven't started looking at the production side yet, but I am going to assume it has something to do with the env files in cookiecutter

Thanks again!

EDIT: to add context, I am working with a fresh cookiecutter project and added a flower container so I can monitor all the workers.

[mjj55409]

It's this bit that's the problem locally:

The main problem is that the custom user running the container needs r/w permissions on the mounted app directory.

You are only using a mounted directory locally.

[amcorreia]

EDIT: to add context, I am working with a fresh cookiecutter project and added a flower container so I can monitor all the workers.

@cjvanderlinden
Can you share your dockerfile to add flower?

[sfdye]

@amcorreia

This is what I did to enable flower in my project, inlocal.yml add:

flower:
    <<: *django
    ports:
      - "5555:5555"
    depends_on:
      - redis
      - postgres
      - celeryworker
      - celerybeat
    command: /start-celeryflower.sh

And then in your compose/local/django/flower add a start.sh

#!/usr/bin/env bash

set -o errexit
set -o pipefail
set -o nounset
set -o xtrace

celery -A scantist.taskapp flower

Lastly, in django's Dockerfile

COPY ./compose/local/django/celery/flower/start.sh /start-celeryflower.sh
RUN sed -i 's/\r//' /start-celeryflower.sh
RUN chmod +x /start-celeryflower.sh

[amcorreia]

@sfdye
Thank you.

[westurner]

C_FORCE_ROOT (to unserialize potentially executable pickles as root; even in a container) is a bad solution.

When I add either of these to local.yml:

• user: 1000
• user: django

I get:

celeryworker_1  | Postgres is unavailable - sleeping
celeryworker_1  | Postgres is unavailable - sleeping
celeryworker_1  | Postgres is unavailable - sleeping

Shouldn't there be an additional user with only read access to the app?
(Whereas the django user can overwrite the app which is owned by django).

IDK why postgres would need different permissions if it's configured to use network sockets?

This is local and without untrusted data; but still it's not good to deserialize pickles as root (or really as any user with write permissions to the app).

[adammsteele]

A PR has been merged that instructs celery to use json as the message serialization by default. Celery worker should no longer complain about this security issue, at the expense of not being about to pass arbitrary python objects as arguments to celery tasks.