Kombu: redis: add configuration parameter to prefix all keys used with a custom string

[wetneb]

I use kombu (via celery) with a shared redis instance. The administrators of that instance require me to prefix all the keys I use in the redis server with a custom string. While Celery provides ways to prefix queue names, it seems to me that there is currently no way in Kombu (and hence Celery) to prefix all the redis keys used.

My understanding is that the keys used for channels are defined by keyprefix_queue = '_kombu.binding.%s', which not easily modifiable from Celery (via BROKER_TRANSPORT_OPTIONS). Would it make sense to add a configuration parameter for that? If that sounds like the correct approach, I can try to provide a PR for that.

[georgepsarakis]

@wetneb have you read the Celery documentation section regarding routing? I think that if you set the queue name it will also be reflected in the key pattern.

Please let me know if you need additional feedback.

[wetneb]

@georgepsarakis indeed it is - but technically it isn't adding a prefix to the keys (as they still start with _kombu.binding.), right?

Prefixing redis keys seems to be a fairly widespread practice so I was just thinking that it could make sense for Celery to support that too. But maybe it is just redundant with queue names and you don't want to create confusion with that?

[georgepsarakis]

@wetneb I am not sure I understand your point. Isn't _kombu.binding.<QUEUE_NAME> the Redis key prefix?

[wetneb]

@georgepsarakis yes exactly - what I would like to do is add another custom prefix in front of that. For instance, if both Mary and John use the same redis database, they could want to make sure that all the keys they use are prefixed with different strings. So they would want to use mary._kombu.binding.<QUEUE_NAME> and john._kombu.binding.<QUEUE_NAME> respectively.

In other words, it would be useful if the _kombu.binding.%s pattern could be altered via the broker_transport_options configuration setting.

This quite common practice: for instance, when using Redis with Django for caching, you can specify such a prefix with KEY_PREFIX:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient"
        },
        "KEY_PREFIX": "example"
    }
}

[georgepsarakis]

@wetneb I think I see your point. Would you like to open a Pull Request for this?

[georgepsarakis]

@wetneb is there still interest for this? It does sound as a useful feature.

[wetneb]

@georgepsarakis thanks for the reminder - it's on my todo list!

[georgepsarakis]

@wetneb great! Let me know what your design thoughts and questions are, once you are ready.

[lswjkllc]

@wetneb @georgepsarakis
hello, is this problem solved?

[lswjkllc]

@thedrow
@georgepsarakis
hello, when can this problem be solved?

[wetneb]

This is not solved, my PR did not make it. Feel free to have a go at it!