Celery: Multiple bindings to the same queue: Only the last binding is being created in RabbitMQ

Created on 31 Oct 2016  路  3Comments  路  Source: celery/celery

Checklist

  • [X] I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • [x] I have verified that the issue exists against the master branch of Celery.
software -> celery:3.1.0 (Cipater) kombu:3.0.37 py:2.7.10
            billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:redis://localhost:6379/0

CELERY_QUEUES:
    (<unbound Queue Test_Queue -> <unbound Exchange test_exchange(topic)> -> myapp.tasks.some_tasks.#>,
 <unbound Queue Test_Queue -> <unbound Exchange test_exchange(topic)> -> myapp.tasks.more_tasks.#>)
RABBITMQ_MANAGEMENT_URL: 'localhost:15672'
CELERY_IMPORTS:
    ('myapp.tasks.some_tasks', 'myapp.tasks.more_tasks')
CELERY_ROUTES:
    (<myapp.routers.DefaultRouter object at 0x1077952d0>,)
BROKER_URL: 'amqp://guest:********@localhost:5672//'
CELERY_RESULT_BACKEND: 'redis://localhost:6379/0'

Steps to reproduce

  1. Create 2 bindings on a topic exchange with different routing keys but point them to the same queue in CELERY_QUEUES.
CELERY_QUEUES = (
    Queue("Test_Queue", default_exchange, routing_key="myapp.tasks.some_tasks.#"),
    Queue("Test_Queue", default_exchange, routing_key="myapp.tasks.more_tasks.#"),
)
  1. Instantiate a celery worker
  2. Navigate to the RabbitMQ admin console http://localhost:15672/
  3. Click onto the Queues tab, then the queue you created in your settings
  4. See that only 1 binding exists
    screen shot 2016-10-31 at 5 52 03 pm

Expected behavior

  1. There should be a second binding with the routing key myapp.tasks.some_tasks.#
  2. This can be replicated in the celery amqp console:
celery amqp
queue.bind Test_Queue test_exchange Key1
queue.bind Test_Queue test_exchange Key2

Actual behavior

  1. Only the last binding of a given queue name is created in rabbitmq

Most helpful comment

CELERY_QUEUES is a mapping from queue_name to queue, so since you are defining two queues
with the same name, only one of them will be used.

To define a queue with multiple bindings in Kombu you should use:

from kombu import Exchange, Queue, binding
default_exchange = Exchange('default', type='direct')

CELERY_QUEUES = (
    Queue('Test_Queue', [
         binding(default_exchange, routing_key='myapp.tasks.some_tasks.#'),
         binding(default_exchange, routing_key='myapp.tasks.more_tasks.#'),
    ]),
)

All 3 comments

CELERY_QUEUES is a mapping from queue_name to queue, so since you are defining two queues
with the same name, only one of them will be used.

To define a queue with multiple bindings in Kombu you should use:

from kombu import Exchange, Queue, binding
default_exchange = Exchange('default', type='direct')

CELERY_QUEUES = (
    Queue('Test_Queue', [
         binding(default_exchange, routing_key='myapp.tasks.some_tasks.#'),
         binding(default_exchange, routing_key='myapp.tasks.more_tasks.#'),
    ]),
)

Great! Just tested it and both bindings have been instantiated. Is this worth adding to the routing docs? I can create a pull request sometime tonight or tomorrow

Sure! That would be a great idea to have it in the docs :)

Was this page helpful?
0 / 5 - 0 ratings