Redis: connection pool timeout

[Netherdrake]

I am using Redis as a caching layer for long running web services. I initialize the connection like so:

var (
    Queues  *redis.Client
    Tracker *redis.Client
)

func Connect(url string) {
    // cut away redis://
    url = url[8:]

    // connect to db #0
    Queues = redis.NewClient(&redis.Options{
        Addr:     url,
        Password: "",
        DB:       0,
    })

    _, err := Queues.Ping().Result()
    if err != nil {
        panic(err)
    }

    // connect to db #1
    Tracker = redis.NewClient(&redis.Options{
        Addr:     url,
        Password: "",
        DB:       1,
    })

    _, err = Tracker.Ping().Result()
    if err != nil {
        panic(err)
    }
}

Abeit in an upcoming patch (sysadmin is deploying a Redis cluster) it will be like so:

var (
    Cluster *redis.ClusterClient
)

func ConnectCluster(cluster, password string) {
    addresses := strings.Split(cluster, ",")
    Cluster = redis.NewClusterClient(&redis.ClusterOptions{
        Addrs: addresses,
        // Password: password,
    })

    _, err := Cluster.Ping().Result()
    if err != nil {
        panic(err)
    }
}

The above code gets run once when service boots up in main.go and the *redis.ClusterClient is being used for the lifetime of the process.

I realize there is an iherent problem with this approach, which is manifesting itself in connections timing out after a few days, and crashing the application:
redis: connection pool timeout
See logs here

Please advise, what would be a proper approach to use go-redis in this situation?

[vmihailenco]

That error can mean 2 things:

• Redis is busy doing some expensive work (unlikely)
• you use PubSub or Multi and don't close it correctly (multi.Close() when multi is not needed any more) so connection is not returned to the pool

[Netherdrake]

So the code is correct?

Could it it be the scheduled backups where dumps of Redis are done in background (they fail sometimes too)?

[Netherdrake]

Also, is there any way to disable panics on redis: connection pool timeout. I would much rather handle this in application rather than have the whole server blow up.

[dim]

There actually shouldn't be any panics. It could be a mistake on your end,
could you share some code?

[Netherdrake]

I have just pushed a cluster in production, so will see if it keeps crashing now.

The old way was, in main.go, Connect would run creating global variables such as Queues *redis.Client.

Then from various goroutines, calls would be made such as Queues.Lpush, Queues.Exists, Queues.Get, etc etc..

I don't know what part of code panics, the logs don't contain that info.

[Netherdrake]

Ok, we've deployed the cluster and the error is persisting. The application keeps crashing with same error redis: connection pool timeout.

[Netherdrake]

Here is how my code works. This is redis.go:

package rds

import (
    "fmt"
    "strings"

    "gopkg.in/redis.v3"
)

var (
    Cluster *redis.ClusterClient
)

func ConnectCluster(cluster, password string) {
    addresses := strings.Split(cluster, ",")
    Cluster = redis.NewClusterClient(&redis.ClusterOptions{
        Addrs: addresses,
        // Password: password,
    })

    _, err := Cluster.Ping().Result()
    if err != nil {
        panic(err)
    }
}

This is main.go:

func main() {
    // -production flag
    flag.Parse()

    // get env variables
    GlobalConfig = config.NewGlobalConfig()

    // connect to MongoDB
    mongo.Init(GlobalConfig.MongoURL)

    // connect to Redis
    rds.ConnectCluster(GlobalConfig.RedisCluster, GlobalConfig.RedisPassword)

    // start the workers
    StartWorkers()

    // main functions
    http.HandleFunc("/X", XHandler)...

This is how I use redis in http Handler goroutines and workers:

        _, err = rds.Cluster.SetNX(key, "true", time.Duration(globalSettings.Server.ExclusivityPeriod)*time.Hour).Result()
        if err != nil {
            log.Println(err)
        }

This is where the error connection pool timeout comes to life.

The redis.v3 package promotes sharing connection between goroutines, yet it is incapable of actually doing so.

I guess my options are:
1.) Reboot the servers every 2 hours.
2.) Rewrite the code so that NEW redisCluster connection is made for EACH goroutine (resource expensive?)

Hopefully its just me being silly and using the library wrong. (easy fix)

Please advise.

[Netherdrake]

I have bumped the options and deployed, lets see if this works:

    Cluster = redis.NewClusterClient(&redis.ClusterOptions{
        Addrs:        addresses,
        PoolSize:     1000,
        PoolTimeout:  2 * time.Minute,
        IdleTimeout:  10 * time.Minute,
        ReadTimeout:  2 * time.Minute,
        WriteTimeout: 1 * time.Minute,
        // Password: password,
    })

[dim]

So yes, the default pool size is rather small. Based on your snippet, you
use one connection per http request, which means your pool size must match
the number of your concurrent http connections. I think, 1000 seems a
little high, but it's really only the maximum cap. It shouldn't matter, as
the client opens new connections on demand.

I will submit a PR to allow applications to instrument the pool size, this
seems to be quite relevant.
On 27 Nov 2015 7:10 am, "Netherdrake" [email protected] wrote:

I have bumped the options and deployed, lets see if this works:

Cluster = redis.NewClusterClient(&redis.ClusterOptions{
    Addrs:        addresses,
    PoolSize:     1000,
    PoolTimeout:  2 * time.Minute,
    IdleTimeout:  10 * time.Minute,
    ReadTimeout:  2 * time.Minute,
    WriteTimeout: 1 * time.Minute,
    // Password: password,
})

—
Reply to this email directly or view it on GitHub
https://github.com/go-redis/redis/issues/195#issuecomment-160056781.

[vmihailenco]

I believe that client with default options can easily serve 1000 concurrent connections without pool timeouts if Redis processes requests fast (within 1-5 ms). I don't mind to bump default PoolSize to 100, but I doubt it will fix this issue...

Meanwhile I will try to run some long-running tests on Redis cluster...

[vmihailenco]

@Netherdrake I think I fixed this issue in master. Please try to upgrade.

[Netherdrake]

Ok, so if I upgrade by pulling down fresh gopkg.in/redis.v3, that will include latest patch?

Edit: Nevermind, it will. https://github.com/go-redis/redis/releases/tag/v3.2.17

I will let you know if I have any issues.

[zhangruiskyline]

Hi, For redis connection pool, every time we perform some actions, like GET, do we need to manually close connection and return connection to pool, or it will be handled by client?

[vmihailenco]

No, you don't have to manually manage connections - go-redis does it automatically.