Fasthttp: "no free connections available to host" while trying to hit high concurrent requests

[reoxey]

Hi,
Was trying the following piece of code.

The idea here is, concurrent requests are received from the users in the main() and those requests further sent to endpoints in req() concurrently. Each request in main() is sent to 5 endpoints concurrently. This code is just to check max QPS handled at the server.

This code fails at higher concurrency.

$ wrk -t5 -c100 -d60s 'http://xxx.xxx.xxx.xxx:4000/o'
After running wrk, no free connections available to host appear in the error log.

I understood what @valyala has explained in #112. But ctx.Request.ResetConnectionClose() is no longer seems to be present in the package. Also, couldn't find any alternative.

var c *fasthttp.Client
var wLog *os.File

func main() {
    runtime.GOMAXPROCS(runtime.NumCPU())
    fmt.Println("Start test")
    const n = 5

    c = &fasthttp.Client{
        ReadTimeout: 400 * time.Millisecond,
        WriteTimeout: 400 * time.Millisecond,
    }
    cLog, _ := os.OpenFile("test.log", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666)
    wLog, _ = os.OpenFile("warn.log", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666)
    log.Print(":4000")

    e := fasthttp.ListenAndServe(":4000", func(ctx *fasthttp.RequestCtx) {
        start := time.Now()
        done := make(chan bool)

        for _ = range [n]int{}{
            go req(done)
        }
        for _ = range [n]int{}{
            <- done
        }

        t := color.Bold(time.Since(start).Round(time.Millisecond))
        L.LogIt(cLog, map[string]string{
            "T": t,
        })
        fmt.Fprintf(ctx, "ok")
    })
    if e != nil {
        fmt.Println(e)
    }
}

func req(done chan bool) {

    rq := fasthttp.AcquireRequest()
    rp := fasthttp.AcquireResponse()
    defer func(){
        fasthttp.ReleaseRequest(rq)
        fasthttp.ReleaseResponse(rp)
    }()
    rq.SetRequestURI("https://some.api.endpoint/")

    //rq.Reset()  // <- Did not help
    //rq.ResetBody() // <- same here
    //rq.ResetConnectionClose() // <- No longer exists as answered in issue #112 by @valyala

    if e := c.DoTimeout(rq, rp, 400 * time.Millisecond); e != nil {
        L.LogIt(wLog, map[string]string{
            "E": color.Red(e),
        })
    } else {
        fmt.Println(rp.Body())
    }
    done <- true
}

So, how do I fix this?

Thanks.

[erikdubbelboer]

Since you don't set Client.MaxConnsPerHost fasthttp will use DefaultMaxConnsPerHost which is 512.

If your https://some.api.endpoint/ doesn't return quickly enough it is possible that there are already 512 connections open in which case you will see this error.

Keep in mind that c.DoTimeout will return after the timeout but the connection will still be open until a response is received (even if that response takes longer than the timeout).

[erikdubbelboer]

You could try to improve this by increasing Client.MaxConnsPerHost. But at some point you're going to have to increase the response time of https://some.api.endpoint/ as well I guess (assuming you have access to this service as well).

[reoxey]

Hi @erikdubbelboer , Is it possible to kill the connection after the endpoint goes beyond the Timeout set in the c.DoTimeout ?

[reoxey]

Also, I tried MaxConnsPerHost. It solved that problem about no free connections.

c = &fasthttp.Client{
        MaxConnsPerHost: 10000,
        ReadTimeout: 400 * time.Millisecond,
        WriteTimeout: 400 * time.Millisecond,
    }

Thanks

[erikdubbelboer]

Setting the ReadTimeout should already be enough to kill the connection after that time. Unless the server streams a big response slowly (since it's the time fasthttp will wait after the previous byte received).

Since you are setting a low ReadTimeout you shouldn't have to use .DoTimeout(. I would suggest you try using .Do( instead.
In your benchmark you are only making 100 connections, which should result in 500 requests, which should be lower than the 512 default limit. But I'm guessing there was some overlap causing the no free connections available to host error. Try using .Do( and setting MaxConnsPerHost to 1000 for example. That should in theory work fine.

[reoxey]

Well, I did try with ReadTimeout and .Do(. The total time for that was going above 400ms, like 600ms. I was expecting it to stop processing the request around 400ms.
Adding .DoTimeout makes it more time abiding which takes max 410ms.
I tried with concurrency of 5000, which gives me a pretty good results of 10k per sec with latency of 450ms.
The clients at the front end expect the results within 500ms.

[reoxey]

Also, requests which hits the endpoints which I have set as 5 are not same domain in the real world. They can be more or less than 5. They have a window of 400ms to reply with a response.

[erikdubbelboer]

The ReadTimeout will go into affect after the request has been written. That's why a timeout might take 600ms for the whole request.

.DoTimeout( just does this last 200ms in a different goroutine and returns earlier because of that.

[theGeekPirate]

@reoxey As a quick side note, you haven't needed runtime.GOMAXPROCS(runtime.NumCPU()) since Go 1.5 (2015/08/19, see https://golang.org/doc/go1.5#runtime).

[chanjarster]

I think it will be better if fasthttp provides a blocking behavior when ErrNoFreeConns happens.
For example, introduce MaxConnWaitTimeout and MaxConnWaitQueue options:

1. if MaxConnWaitTimeout is not set and there are no free connections, just return ErrNoFreeConns. Or, block the request until MaxConnWaitTimeout elapse(return ErrNoFreeConns) or got a connection
2. if MaxConnWaitQueue is not set, the number of blocked requests will be unlimited. Or, returns ErrConnWaitQueueFull when exceeds the number.

[erikdubbelboer]

@chanjarster someone made a pull request for this feature but never bothered to finish it: https://github.com/valyala/fasthttp/pull/668

[chanjarster]

@erikdubbelboer I created a PR #764 , looking forward to your suggestion

[chanjarster]

@reoxey Since PR #764 has been merged, you can try it.

[Fenny]

I think this issue can be closed since https://github.com/valyala/fasthttp/pull/764 is merged.