Cookiecutter-django: Look to implement django-q

[scwall]

Description

https://github.com/Koed00/django-q
https://django-q.readthedocs.io/en/latest/

Hi I also have a suggestion about a library for asynchronous processing that I use for that ease and speed. django-q. I think it would be nice to have her as a second choice in celery. Possibility to use the orm as a brooker
and it allows the addition of repetitive task type

Rationale

A multiprocessing distributed task queue for Django

Features:

• Multiprocessing worker pool
• Asynchronous tasks
• Scheduled and repeated tasks
• Encrypted and compressed packages
• Failure and success database or cache
• Result hooks, groups and chains
• Django Admin integration
• PaaS compatible with multiple instances
• Multi cluster monitor
• Redis, Disque, IronMQ, SQS, MongoDB or ORM
• Rollbar and Sentry support
-

Use case(s) / visualization(s)

Creating Tasks
Use async_task from your code to quickly offload tasks:

from django_q.tasks import async_task, result

# create the task
async_task('math.copysign', 2, -2)

# or with a reference
import math.copysign

task_id = async_task(copysign, 2, -2)

# get the result
task_result = result(task_id)

# result returns None if the task has not been executed yet
# you can wait for it
task_result = result(task_id, 200)

# but in most cases you will want to use a hook:

async_task('math.modf', 2.5, hook='hooks.print_result')

# hooks.py
def print_result(task):
    print(task.result)

# Use the schedule function
from django_q.tasks import schedule

schedule('math.copysign',
         2, -2,
         hook='hooks.print_result',
         schedule_type=Schedule.DAILY)

# Or create the object directly
from django_q.models import Schedule

Schedule.objects.create(func='math.copysign',
                        hook='hooks.print_result',
                        args='2,-2',
                        schedule_type=Schedule.DAILY
                        )

# Run a task every 5 minutes, starting at 6 today
# for 2 hours
import arrow

schedule('math.hypot',
         3, 4,
         schedule_type=Schedule.MINUTES,
         minutes=5,
         repeats=24,

         next_run=arrow.utcnow().replace(hour=18, minute=0))

[browniebroke]

Thanks for bringing this up, it looks like an intersting library.

However, looking at their build matrix, it's currently not tested against the version of Django we use (2.2 currently). Also, having never used this in production at this point, I wouldn't be able to vouch for it and include it in this project, but maybe another maintainer is.

I'll leave this here for a bit.

[scwall]

Hello, he just released the release yesterday, it looks good. :D
https://github.com/Koed00/django-q/releases/tag/v1.0.2

V1.0.2
@Koed00 Koed00 released this 20 hours ago

Fixes deprecated Arrow interface issues.
Tests with Django 2.2.4 and 1.11.23
SQL server fix @wgordon17
Circular import fix @lamby
MySQL fix for no timezone configs @maerteijn
Loads of timeout and concurrency fixes from @jannero

Production :

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": env("REDIS_URL"),
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            # Mimicing memcache behavior.
            # http://niwinz.github.io/django-redis/latest/#_memcached_exceptions_behavior
            "IGNORE_EXCEPTIONS": True,
        },
    }
}
# DJANGO-Q
# redis defaults
Q_CLUSTER = {
    'name': 'DJRedis',
    'workers': 4,
    'timeout': 90,
    'django_redis': 'default'
}

and development :

Q_CLUSTER = {
    'name': 'DjangORM',
    'workers': 10,
    'timeout': 90,
    'retry': 120,
    'queue_limit': 50,
    'orm': 'default'
}

You can use the connection of the django cache to be re-disposed directly without having to recreate a second connection to it

[janneronkko]

Note that the example configuration for production has timeout set to 90 which is bigger than the Django-Q default for retry (60).

If you start a task that takes 61 seconds, the task will be retryed before it completes. In fact, Django-Q might retry the task after the task has been running, for example, 5 seconds.

See the following for reference:

• https://github.com/Koed00/django-q/commit/d3f42b46d4694a1282b68b18811f8b9bdd2345b9
• https://django-q.readthedocs.io/en/latest/configure.html#timeout
• https://django-q.readthedocs.io/en/latest/configure.html#retry

Also see https://github.com/Koed00/django-q/issues/344 as the behaviour I noted above is in deep within Django-Q's implementation.

Django-Q is a nice, easy to use library but you might get hit by the above issue. I was hit by that quite hardly earlier this year.

[scwall]

Thank's. I must admit, I didn't pay attention to that subtlety. I was able to see what kind of problem I might encounter in the future.

[browniebroke]

Thanks for the proposal, but I think we'll stick to Celery for the time being. Supporting 2 frameworks for asynchrnous processing feel a bit too much at this point, I think Cookiecutter Django should be opiniated and choose a solution for this. For now, Celery it is.