Redis: Pubsub Connection Pooling

[tpavs]

So I am trying to understand if and how connection pooling for this pubsub client is working.

I am using this test script

package main

import (
    "flag"
    "fmt"
    "github.com/go-redis/redis"
)

var redisAddr = flag.String(
"redis-caches",
"redis-cache:6379",
"comma separated list of redis caches",
)

func main() {
    flag.Parse()
    client := redis.NewClient(&redis.Options{ Addr: *redisAddr })
    for i := 0; i < 100000000; i++ {
        client.Subscribe(fmt.Sprintf("S%d", i))
        println(fmt.Sprintf("Subscribing to %d %d", client.PoolStats().FreeConns, client.PoolStats().TotalConns))
    }
}

It appears to me like the pool just keeps allocating connections.

I also tried setting it up where I do a Subscribe() and then just use the returned pubsub value which appears to work but that means that basically all traffic will go through 1 connection to redis. Ideally I would like to be able to use N connections to redis but have that N be limited on each box to make sure that I don't drown redis or starve the box of connections. Is this not possible?

I saw this old issue https://github.com/go-redis/redis/issues/459 which discussed this and there was a test in previous versions. The test in question appears to be mia on master/in the 6.12.0 release. Should I be building against that release or should I drop back to v5?

[vmihailenco]

PubSub intentionally bypasses connection pool limits, because PubSub connections can't be safely reused.

[tpavs]

Thanks for the quick response. Appreciate it I will continue with my single connection approach.

for anyone who is reading in the future

pubsub := client.Subscribe()
then use pubsub for all subscriptions and unsubscriptions

Thanks!

[rush24]

PubSub intentionally bypasses connection pool limits, because PubSub connections can't be safely reused.

Could you please explain more in detail? that means it's useless when we setting pool_size of the client for pubsub, which may making redis server conns out of limit, the problem had happended in our app. so what could be the cause of this? is there a best practice for using pubsub conn? @vmihailenco

[vmihailenco]

Usually apps create few PubSub subscriptions and use them until app exits. In such case connection pooling is not needed.

If you really need to create and close hundreds of PubSubs then you also need to somehow limit the number of active PubSubs in your app. See https://github.com/golang/go/wiki/RateLimiting

[stemcd]

The Java client (Jedis) supports multiple pubsub channels per client connection.

The c# implementation actually calls this out as a feature https://github.com/StackExchange/StackExchange.Redis/issues/872

Why goes the go-redis implementation require a new connection per pubsub channel?

[vmihailenco]

@stemcd it does not, it requires a connection per PubSub instance.

[stemcd]

Thanks for the quick response :) Feels like either you misunderstood my point or I am misunderstanding how Redis works. In Redis "the global pubsub_channels variable is logically a Map>, and each client’s subscription set is a Set". See this for reference https://making.pusher.com/redis-pubsub-under-the-hood/

Effectively this means that a single client connection can listen to multiple channels. Are you saying that this is not possible with the current implementation of go-redis?

I have just tested this in the Java Jedis implementation with one client connection subscribed to two channels. Redis info reports connected_clients:1 and pubsub_channels:2. If I do the same in go-redis I get connected_clients:2 and pubsub_channels:2

[vmihailenco]

Most likely you create two PubSub instances instead of passing 2 channels to the single instance - Subscribe("chan1", "chan2").

[stemcd]

Thanks, but your suggestion implies that you know all your channels at the moment of subscribing, so it is a static list of channel names per connection. In my use case I wish to dynamically manage a very large number of very quiet channels using a small number of Redis connections.

I have just done a Proof of Concept with Radix and monitored the results with the redis info command. I was able to dynamically make thousands of concurrent Subscribe and Unsubscribe using a single redis channel. pubsub_channels peaked at about 250 but connected_clients stayed at 1. For reference https://godoc.org/github.com/mediocregopher/radix#PubSubConn My next goal is to investigate their pooling mechanism to balance the channels across a pool of active connections.

I am thinking that I will need to switch my application to use Radix because I can find no way to do dynamic subscription with go-redis. I can't use Subscribe("chan1", "chan2") as you said because the list of channel names is dynamic. Each call to Subscribe(...) creates a new connection so this is too expensive.

Thanks for your help, but I suspect this due to the nature of the current implementation of go-redis and is a blocker for me for now. :/

[vmihailenco]

You can subscribe later as needed - https://godoc.org/github.com/go-redis/redis#PubSub.Subscribe. You probably confuse Client.Subscribe and PubSub.Subscribe.

[stemcd]

Thanks again for your help. I didn't have this confusion, but it does help explain why we are not understanding each others perspective. This method assumes every Pub Sub Client has exactly 1 Go Channel, 1 Redis Connection, and is Subscribed to N Redis Pub Sub Channels.

The pattern I am looking for, and have implemented in other libraries is 1 Redis Connection has a number of consumers. Each consumer has 1 Go Channel and 1 Redis pubsub Channel. This means that consumers can arrive dynamically and subscribe to a to new Redis Channel using the existing small Redis connection pool rather than make an expensive Redis connection for every Consumer.

One advantage of the go-redis implementation is that the assumptions mean that the client library can manage the Redis Connection and so can reconnect after disconnects etc. This is a reasonable trade off to make but the alternative is to have a weaker relationship but support many more concurrent dynamic consumers.

[vmihailenco]

Okay, sounds like you want a separate Go channel for every subscribed Redis channel, but go-redis PubSub provides single Go channel for all messages received from single PubSub.

If that is true I would

• consider if you really want a separate Go channel for each consumer - channels are useful when they are shared between several consumers
• if you really need Go channels you can route messages from single channel to arbitrary number of channels in user space
• or you can write something without using channels

for msg := range pubsub.Channel() {
    consumer := getConsumer(msg.Channel) // or getConsumers if there can be multiple consumers
    err := consumer.ProcessMessage(msg)
}