Fasthttp: Customize Host Header

[sschepens]

Almost every Request library supports changing the Host header, for example for Proxying, once connects to a given IP, but specifies another Host for Forwarding request.
This is currently not possible with fasthttp setting request URI and then Host Header makes Client ignore the Header and send whatever was set as Host on the RequestURI.

For example:

req := fasthttp.AcquireRequest()
req.SetRequestURI("http://PROXY_IP:PROXY_PORT")
req.Header.SetHost("www.google.com")

This should be totally possible, what do you think?

[valyala]

Yes, overriding host header should be possible. This looks like a bug. Will look into it.

[valyala]

Further investigation showed that this is not a bug, but a feature :)

See Client.Do documentation:

Client determines the server to be requested in the following order:

- from RequestURI if it contains full url with scheme and host;
- from Host header otherwise.

The RequestURI in your case contains full uri with scheme and host, so the client ignores Host header when determining the server to connect to. But the client sends the provided Host header to the server. This allows sending requests to distinct virtual hosts on a single physical host.

I'd recommend using HostClient istead of Client for upstream (aka reverse) proxying. It load balances requests among the given set of upstream hosts set in HostClient.Addr irregardless of Host set in headers or in RequestURI.

If you still want using Client instead of HostClient (for instance, for general proxy implementation), then use specially added Request.SetHost method. It overrides the host from RequestURI.

[sschepens]

@valyala I tried with HostClient but it didn't seem to work, It seemed to ignore HostClient.Addr

[valyala]

@sschepens , could you provide a small reproducible test case for further investigation?

[ldemailly]

@valyala I have the same issue (not figured out how to set Host header yet connect to what I give in URI) :

type fastClient struct {
    client *fasthttp.Client
    req    *fasthttp.Request
    res    *fasthttp.Response
}

// NewFastClient wrapper for the fasthttp library
func NewFastClient(url string) Fetcher {
    cli := fastClient{
        client: &fasthttp.Client{},
        req:    fasthttp.AcquireRequest(),
        res:    fasthttp.AcquireResponse(),
    }
    cli.req.SetRequestURI(url)
    cli.client.ReadBufferSize = 16384
    if hostOverride != "" {
        log.Printf("Setting host to %s", hostOverride)
        cli.req.SetHost(hostOverride)
    }
    for h := range extraHeaders {
        cli.req.Header.Set(h, extraHeaders.Get(h))
    }
    return &cli
}

func (c *fastClient) Fetch() (int, []byte, int) {
    if err := c.client.Do(c.req, c.res); err != nil {
        log.Printf("Fasthttp error %v", err)
        return 400, nil, 0
    }
    // TODO: Header.Len() is number of headers not byte size of headers
    return c.res.StatusCode(), c.res.Body(), c.res.Header.Len()
}

when I call SetHost it tries to connect to that Host instead of the one in the URI:

2017/07/17 19:12:49 http.go:605: Setting host to blah
2017/07/17 19:12:49 http.go:616: Fasthttp error lookup blah: no such host
Aborting because of error 400 for http://localhost:8081

I tried with req.Header.Set("host",...) and req.Header.SetHost() and those just get ignored/replaced by the one from the URI:

I'm calling with "http://localhost:8081/" as the URI and have netcat -l 8081 running:

GET / HTTP/1.1
User-Agent: istio/fortio-0.1
Host: localhost:8081
Foo: bar
H1: val1

my other headers are set but what's the trick for Host: ?
(I suppose I could flip it around, hack the url to replace it by the Host header and then set Host to the original one... but surely there is a better way to just set whatever I want in the http headers before the request is sent)

[ldemailly]

Note : I tried setting the url to http://blah and the req.SetHost to be localhost:8081 (ie flip the 2 around, which is not how curl and wrk and other tools work but...) and it doesn't work either, it does connect to localhost:8081 but also sets Host: header to localhost:8081

HostClient seems a lot more complicated than what I want so I'd really like a simple solution with Client