Fasthttp: apache + fasthttp errors

[lunemec]

Hello,

I'm experiencing similar issue to #112 but your suggestion doesn't work. I have fasthttp proxy in front of apache server. Upon recieving request to fasthttp I create new request crafted for apache (upstream) and depending upon its response, fasthttp responds.

func handler(ctx *fasthttp.RequestCtx, _ fasthttprouter.Params) fasthttprouter.Handle {
    upstreamRequest := fasthttp.AcquireRequest()
    upstreamRequest.Header.ResetConnectionClose()
    // Other request data set here.
    fasthttp.DoTimeout(upstreamRequest, response, timeout)
}

But even when I added ResetConnectionClose(), I stil get these errors:

the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection

Here is complete traceback:

goroutine 28 [running]:
runtime/debug.Stack(0x9b3538, 0x6c9b95, 0x4)
    /usr/local/Cellar/go/1.7.1/libexec/src/runtime/debug/stack.go:24 +0x87
lingea-validator/vendor/github.com/mgutz/logxi/v1.(*HappyDevFormatter).getLevelContext(0xc4200e3000, 0x3, 0xc420212690, 0x0, 0x0, 0x0, 0x0, 0xc42015e2e8, 0x5)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/main.go:136 +0x777
lingea-validator/vendor/github.com/buaazp/fasthttprouter.(*Router).Handler(0xc42000cc60, 0xc42010b680)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/vendor/github.com/buaazp/fasthttprouter/router.go:331 +0xb82
lingea-validator/vendor/github.com/buaazp/fasthttprouter.(*Router).Handler-fm(0xc42010b680)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/main.go:207 +0x4c
lingea-validator/vendor/github.com/valyala/fasthttp.TimeoutHandler.func1.1(0xc420010e70, 0xc42010b680, 0xc4200711a0, 0xc4201ac360)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/vendor/github.com/valyala/fasthttp/server.go:313 +0x42
created by lingea-validator/vendor/github.com/valyala/fasthttp.TimeoutHandler.func1
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/vendor/github.com/valyala/fasthttp/server.go:316 +0x168

What I think is causing problems, is Apache, even when Keepalive enabled, has a request limit (default 100) on how many requests are handled with one TCP connection. Also, apache kills some of its worker processes upon reaching some other connection limit. And from documentation, it keeps keepalive open for 5s by default.

I can disable keepalive on apache side, but I'd rather disable it on fasthttp side. Can this be solved somehow?

Oh, and one thing to mention, this does happen with no load. I make the incomming request to fasthttp with telnet - 0.5 req/s :)

Thank you!

[lunemec]

Update: disabling keepalive on apache (upstream) did the trick. But I'd rather use the keepalive since it is between 2 servers, it is waste of CPU to create new TCP connections :)

[valyala]

What I think is causing problems, is Apache, even when Keepalive enabled, has a request limit (default 100) on how many requests are handled with one TCP connection. Also, apache kills some of its worker processes upon reaching some other connection limit. And from documentation, it keeps keepalive open for 5s by default.

I think you are correct - Apache just closes incoming connection after KeepAliveTimeout (5s by default).

fasthttp client detects such a closed connection only after an unsuccessful attempt to read the next response from the connection. This leads to the error:

the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection

Unfortunately the current fasthttp client architecture cannot reliably detect such a case in order to immediately throw out closed connection from the pool.

Possible workarounds:
1) Just retry sending the request on this error.
2) Set HostClient.MaxConnDuration to value smaller than the KeepAliveTimeout in Apache configs. For instance, set it to 4s for default KeepAliveTimeout=5s. This way fasthttp will close the connection before Apache does this.
3) Close the connection after each request to Apache by calling Request.SetConnectionClose on the request before passing it to Apache.
4) Try using PipelineClient. It automatically detects and re-opens closed connections. But beware of head of line blocking problem.

[lunemec]

Thank you for the reply.
I tried all solutions, but only one works reliably - disabling KeepAlive on Apache server.

When using MaxConnDuration, you run into problems of having different timer start on apache and fasthttp and again, apache closes connection first and fasthttp throws error.
I tried this:

upstreamRequest := fasthttp.AcquireRequest()
err := fasthttp.DoTimeout(upstreamRequest, response, timeout)
upstreamRequest.SetConnectionClose()

so the connection would close and not return back to the pool, but it doesn't help, sometimes fasthttp throws the connection:closed error.

I'll take a look at PipelineClient, and implement something to retry the request based on that code. I'll let you know if it helps.

Thank you.

[lunemec]

I tried using PipelineClient from current master, but after exactly 5s of incomming requests I get this as error:

10:23:21.162208 ERR ~ Error calling client. error: EOF
   uuid: 0fc8816f-fe2f-4549-8023-7e4e60d806a7
goroutine 969 [running]:
runtime/debug.Stack(0x9cc898, 0x6d4335, 0x4)
    /usr/local/Cellar/go/1.7.1/libexec/src/runtime/debug/stack.go:24 +0x87
lingea-validator/vendor/github.com/mgutz/logxi/v1.(*HappyDevFormatter).getLevelContext(0xc4200e3000, 0x3, 0xc42000c180, 0x0, 0x0, 0x0, 0x0, 0xc4201682e8, 0x5)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/main.go:141 +0x77f
lingea-validator/vendor/github.com/buaazp/fasthttprouter.(*Router).Handler(0xc42016b6e0, 0xc42027d680)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/vendor/github.com/buaazp/fasthttprouter/router.go:331 +0xb82
lingea-validator/vendor/github.com/buaazp/fasthttprouter.(*Router).Handler-fm(0xc42027d680)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/main.go:213 +0x4c
lingea-validator/vendor/github.com/valyala/fasthttp.TimeoutHandler.func1.1(0xc420169850, 0xc42027d680, 0xc420211f80, 0xc420074d80)
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/vendor/github.com/valyala/fasthttp/server.go:312 +0x42
created by lingea-validator/vendor/github.com/valyala/fasthttp.TimeoutHandler.func1
    /Users/lukas.nemec2/Documents/git/Go/src/lingea-validator/vendor/github.com/valyala/fasthttp/server.go:315 +0x168

This is how I construct PipelineClient:

    pipeline := fasthttp.PipelineClient{
        Addr: upstreamAddress,
        MaxConns: 50,
        MaxIdleConnDuration: (time.Duration(4) * time.Second),
        IsTLS: false,

    }

Any ideas?

Thank you!

[valyala]

Just FYI,

upstreamRequest := fasthttp.AcquireRequest()
err := fasthttp.DoTimeout(upstreamRequest, response, timeout)
upstreamRequest.SetConnectionClose()

The SetConnectionClose must be called before calling DoTimeout. This way the client detects Connection: close request header and closes the connection after receiving the response in DoTimeout:

upstreamRequest := fasthttp.AcquireRequest()
upstreamRequest.SetConnectionClose()
err := fasthttp.DoTimeout(upstreamRequest, response, timeout)

As for PipelineClient returning io.EOF, it requires additional investigation.

[lunemec]

Thank you for the response. I fixed the issue by calling the upstream again if the error was ConnectionClosed.