Kombu: Is there a reason for \x06\x16 to be used as priority separator?

[teferi]

It makes it really difficult to query redis from command line.

I tried redis-cli type "queue_name\x06\x163" and many other variants with some sort of escaping, but couldn't make it work.
Also as far as I understand the naming convention in redis is to use ':' as a separator. redis.io

Could we maybe make it configurable via an environment variable or otherwise?

[ask]

I often want to change this, but would prefer to find a way that does not break backwards incompatibility.

There was no convention at the point when this was written, I guess a transport option would be possible
but would much rather have a sensible default.

[teferi]

We could for example check if the separator differs from \x06\x16 and issue a number of rename's in a redis transaction (to avoid concurrency problems)? I'd like to try and figure smth out with that.

btw: we have found a workaround echo 'llen "queue_name\x06\x163"' | redis-cli if someone ever finds this via google

[mlissner]

I just ran into this too, and on a whim googled for \x06\x163. I'm happy I found this ticket.

This is causing problems for me because I have a CeleryThrottle class that can be put into a batch processing loop like so:

throttle = CeleryThrottle(q='some_queue')
for item in really_long_list:
    throttle.maybe_wait()
    process_item.delay(item)

The idea is that it prevents a runaway Celery queue when processing millions of items. It does this by preventing the loop from proceeding until the queue is sufficiently short (100 items by default). Without the throttle, millions of tasks get added to redis all at once, and things go ka-boom.

The way the throttle works is to periodically query redis to see how long the queue is, but with this weirdness going on, I'm not sure I understand what key to query in redis.

Until this is fixed, is there a clear answer to that question? Will the key in redis always be named after the queue or after the queue + \x06\x16?

Thanks.

[georgepsarakis]

@mlissner perhaps getting the current worker state can help you achieving this.

[mlissner]

@georgepsarakis No, that only lets you see what the workers have, not what is in the queue. There's no official way to see what's in the queue that I've been able to find.

[georgepsarakis]

The flower project seems to have a solid implementation of what you want probably, see here.

[mlissner]

@georgepsarakis That's marvelous, thank you very much for this pointer.

[georgepsarakis]

@mlissner you are welcome, hope it helps!

[mlissner]

This raises an issue with the documentation, which says that you can use the following to get the number of tasks in a queue:

$ redis-cli -h HOST -p PORT -n DATABASE_NUMBER llen QUEUE_NAME

We should tweak these docs to mention that that query will only return P0 tasks.

[jobec]

If anyone would like to change this behavior. You can monkey patch it like this (on your own risk 😇 ):

import kombu.transport.redis
kombu.transport.redis.Channel.sep = ":"

Put it before you instantiate your celery app.