fasthttp.Client shows low performance handling TLS compared to http.Client

[mayocream]

Summary

I'm doing benchmarks between HTTP, HTTPS and mTLS.
When using fasthttp.Client to get response through HTTPS from upstream server, its performance drops a lot.

HTTP QPS: 36k
HTTPS QPS: 10k

It drops more serving mTLS: 6k QPS.

I switched to Golang's http.Client, and its QPS is about 30k.

Troubleshooting

There are pprof flamegraph:

Using fasthttp.Client:

It seems to be not reusing TLS connections, and re-perform TLS handshakes.

Using http.Client:

Reproduce

Starting fasthttp.Server accept requests, return 200 OK.

func main() {
    flag.Parse()

    srv := &fasthttp.Server{
        IdleTimeout:        30 * time.Second,
        TCPKeepalive:       true,
        TCPKeepalivePeriod: 30 * time.Second,
        MaxConnsPerIP:      200,
        DisableKeepalive:   *keepAlive,
    }
    var tlsCfg *tls.Config

    switch *mode {
    case "http":
    case "https":
        _ = mTlsConfig()
        tlsCfg = &tls.Config{
            ClientAuth:   tls.NoClientCert,
            Certificates: []tls.Certificate{*cert},
        }
    case "mtls":
        tlsCfg = mTlsConfig()
    default:
        panic("unknown mode")
    }

    ln, err := net.Listen("tcp4", *addr)
    if err != nil {
        panic(err)
    }

    defer ln.Close()

    lnTls := tls.NewListener(ln, tlsCfg)

    srv.Handler = func(ctx *fasthttp.RequestCtx) {
        ctx.SetStatusCode(200)
        ctx.SetBody([]byte("hello"))
    }

    fmt.Println("started.")

    if tlsCfg != nil {
        if err := srv.Serve(lnTls); err != nil {
            panic(err)
        }
    } else {
        if err := srv.Serve(ln); err != nil {
            panic(err)
        }
    }

    select {}
}

And client serving fasthttp.Server to accept ab benchmark requests, pass requests to upstream server.

func main() {
    flag.Parse()

    var normalClient *http.Client
    client := &fasthttp.Client{}

    normalTr := &http.Transport{
        TLSHandshakeTimeout: 5 * time.Second,
        MaxIdleConns:        2000,
        MaxIdleConnsPerHost: 200,
        MaxConnsPerHost:     2000,
        IdleConnTimeout:     20 * time.Second,
        WriteBufferSize:     1024,
        ReadBufferSize:      1024,
    }

    switch *mode {
    case "http":
    case "https":
        mtlsConfig := mTlsConfig()
        tlsConfig := &tls.Config{
            RootCAs: mtlsConfig.RootCAs,
        }
        client.TLSConfig = tlsConfig
        normalTr.TLSClientConfig = tlsConfig
    case "mtls":
        mtlsConfig := mTlsConfig()
        client.TLSConfig = mtlsConfig
        normalTr.TLSClientConfig = mtlsConfig
    default:
        panic("unknown mode")
    }

    if client.TLSConfig != nil {
        client.TLSConfig.ClientSessionCache = tls.NewLRUClientSessionCache(64)
    }

    normalClient = &http.Client{
        Transport: normalTr,
    }

    srv := &fasthttp.Server{
        IdleTimeout:        30 * time.Second,
        TCPKeepalive:       true,
        TCPKeepalivePeriod: 30 * time.Second,
        MaxConnsPerIP:      200,
    }
    srv.Handler = func(ctx *fasthttp.RequestCtx) {
        if *normal {
            resp, err := normalClient.Get("https://" + *server)
            if err != nil {
                panic(err)
            }
            body, err := ioutil.ReadAll(resp.Body)
            if err != nil {
                panic(err)
            }
            ctx.SetStatusCode(resp.StatusCode)
            ctx.SetBody(body)
            return
        }

        req := &ctx.Request
        req.URI().SetHost(*server)
        scheme := "http"
        if *mode == "https" || *mode == "mtls" {
            scheme = "https"
        }
        req.URI().SetScheme(scheme)
        err := client.Do(req, &ctx.Response)
        if err != nil {
            panic(err)
        }
    }

    fmt.Println("started.")

    srv.ListenAndServe(*addr)

    select {}
}

The server and client TLS certificates are signed by local CA with ECDSA P-256 algo.
The Root and CA certificates are signed with RSA 4096 algo.

Environment

Ubuntu 16.04
8 cores 16g RAM

[gitsrc]

We have encountered similar problems and hope to get help.

Is the configuration incorrect?

Looking forward to the help of the powerful project leader.

[moredure]

I am also highly rely on fasthttp.Client, not experiencing such an issues, but anyway. Any updates?

[erikdubbelboer]

I have build a standalone program to test this: https://gist.github.com/erikdubbelboer/f14b38d258f376dd6d4d47491f32535a

Here is the results when I run it:

% go run mtlstest.go # with fasthttpClient 
86521 14
95866 14
95999 14
96134 14
95562 14
94059 14
% go run mtlstest.go # with httpClient
58288 53
58663 53
57358 53
56933 53
56595 53
55391 53

As you can see fasthttp is much faster here. Maybe there is something wrong in the way you setup you tls.Config. Can you share your full code so I can have a look? Or maybe you can look at my program and copy the way I do things?